Agent Access Index

Whether an AI agent may fetch a URL, on what terms, and at what price. Free, no key, no signup.

Ask it

curl 'https://agentwares-access-index.vercel.app/policy?url=https://example.com/article'

Or point an MCP client at https://agentwares-access-index.vercel.app/mcp. No key, no signup, no payment.

Verdicts

allowed

The origin served a request that said who it was.

allowed_if_identified

The site's written policy permits this fetch. An anonymous request was refused by bot management — which is a refusal with a documented way out, not a decision the publisher made about you.

charged

Access is for sale. The price is quoted when the rail will show one to an unauthenticated request; when it will not, that is said rather than guessed. An x402 quote also carries payable: whether a standard client could actually build the payment from what the 402 said, and which fields are missing when it could not. A price is not an offer — without the scheme, the amount in atomic units, the token’s contract address and its EIP-712 domain, there is nothing to sign.

blocked

The site disallows the agents that would want this page, and offers nothing to buy.

unknown

The origin did not answer, the URL is not there, or the signals conflict.

What this fetches, and how to allow or block it

One request per lookup, to the URL someone asked about, plus that origin's robots.txt and the well-known files that state its terms. No link-following, no crawling, no sitemap walking. Nothing is retained but the verdict and the rule that produced it; no page content is stored, sold, or used to train anything.

To allow it, permit the user-agent below. To refuse it, disallow that token in robots.txt and it will stop, and say so in its answer about your site.

What it will not do

• Solve a CAPTCHA, rotate a proxy, or send a browser user-agent.
• Impersonate another operator's crawler to find out how a site treats it. The per-agent column comes from what the origin declared.
• Fetch a page its own robots.txt rule disallows. That refusal is reported, not worked around.

Every request identifies itself as agent-access-index/1.0 (+https://agentwares-access-index.vercel.app/#what-this-fetches). Operators: what this fetches, and how to allow or block it.

Agents it reports terms for

Only tokens their operators document — the ones a crawler actually sends.

• ClaudeBot — Anthropic
• Claude-User — Anthropic
• Claude-SearchBot — Anthropic
• GPTBot — OpenAI
• OAI-SearchBot — OpenAI
• ChatGPT-User — OpenAI
• Google-Extended — Google
• Google-Agent — Google
• GoogleOther — Google
• PerplexityBot — Perplexity
• Perplexity-User — Perplexity
• CCBot — Common Crawl
• meta-externalagent — Meta
• meta-externalfetcher — Meta
• Amazonbot — Amazon
• Applebot-Extended — Apple
• Bytespider — ByteDance
• MistralAI-User — Mistral
• DuckAssistBot — DuckDuckGo

Machine-readable

• llms.txt
• server.json
• health

agent-access-index 0.1.0