I Read All 30 Official AI Crawler Docs. Only 8 Bots Train Models. The Copy-Paste Blocklist Blocks 25.

TL;DR: No, not if you want to be cited. On August 15, 2026 I read the official crawler docs of 14 AI companies and counted 30 user agents across the 11 that publish any. Only 8 are training crawlers. Another 8 are retrieval crawlers that decide whether you appear in that assistant's answers, and 8 are live fetchers that run when a user asks a question, 6 of which the vendors say on the record may ignore robots.txt anyway. The 4,057-star "block AI bots" robots.txt on GitHub puts 25 of those 30 under a single Disallow: /. Pasting it removes you from ChatGPT search, Perplexity, Claude search, Alexa, Meta AI, DuckAssist and Apple's Spotlight, Siri and Safari results, for the sake of a training opt-out you could have taken in 8 lines.

Who is this for?

The "block the AI bots" argument was written for publishers with licensing leverage, whose archives are worth a negotiation and whose traffic is worth more than the citation. For them, blocking is a bargaining position, though the machinery for actually selling access is thinner than the announcements suggest: three of 36 named RSL supporters serve the license file.

A small site has the opposite problem. Nobody is negotiating with you, and you need to be read to be cited. The advice still arrives as a snippet, and the snippet cannot tell the bot that eats your archive from the bot that quotes you.

What do the vendors actually document?

Twelve of fourteen crawler pages resolved. Cohere's bot table reads "N/A": the company says it operates no crawlers for training data "at this time." ByteDance and xAI publish nothing I could load (x.ai/crawler and docs.x.ai/docs/crawler both 404).

The 11 that document at least one agent:

Company	Training	Retrieval index	Live user fetch
OpenAI	GPTBot	OAI-SearchBot	ChatGPT-User
Anthropic	ClaudeBot	Claude-SearchBot	Claude-User
Google	Google-Extended	(Googlebot; no AI token)	Google-Agent, Google-GeminiNotebook
Perplexity	none documented	PerplexityBot	Perplexity-User
Meta	meta-externalagent	meta-webindexer	meta-externalfetcher
Apple	Applebot-Extended	Applebot	none
Amazon	Amazonbot (mixed)	Amzn-SearchBot	Amzn-User
Mistral	MistralAI-Training	MistralAI-Index	MistralAI-User
Common Crawl	CCBot	none	none
DuckDuckGo	none	DuckAssistBot	none
Allen Institute	AI2Bot	none	none

Eight of those are training-only, eight retrieval, eight live user fetches, plus the mixed Amazonbot. Five more are ads, previews or generic crawling: OAI-AdsBot, meta-externalads, facebookexternalhit, GoogleOther, Google-CloudVertexBot. Thirty.

The vendors are explicit about the cost. OpenAI: sites "opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." Anthropic says disabling Claude-User "prevents our system from retrieving your content in response to a user query." Apple is the sharpest, because Applebot is not an AI bot at all: enabling it "allows website content to appear in search results for Apple users" in Spotlight, Siri and Safari. Disallow it to dodge Apple Intelligence and you have left three products. The training opt-out is Applebot-Extended.

Which of these actually obey robots.txt?

Six of the eight live fetchers say on the record that they may not. Perplexity: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." OpenAI on ChatGPT-User: "Because these actions are initiated by a user, robots.txt rules may not apply." Google on Google-Agent and Google-GeminiNotebook: "Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules." Meta's Meta-ExternalFetcher "may bypass robots.txt because it performs fetches that were requested by the user." Amazon's Amzn-User "may not follow all robots.txt directives."

Two do not. Anthropic says all three of its bots, Claude-User included, "respect 'do not crawl' signals by honoring industry standard directives in robots.txt." Mistral documents MistralAI-User as something robots.txt "governs," no bypass clause.

So the snippet fails twice over: the lines that work are the ones you did not want, and the lines you wanted mostly do not work. The list in question is ai-robots-txt/ai.robots.txt, 4,057 stars and 163 user-agent lines under a single Disallow: /. It blocks 7 of the 8 retrieval crawlers and 7 of the 8 live fetchers.

What are the steps?

1. Read your CDN's bot rules first. Cloudflare, Vercel and Fastly can all 403 these agents at the edge regardless of your file, and Cloudflare's September 15 default splits search, agent and training into separate switches, so the edge is about to make a three-way decision on your behalf. Failure mode: you edit robots.txt, nothing changes, and you conclude AI bots ignore it. They never saw it. Same inherited-default trap I hit reading 24 sites' robots.txt.

2. Answer two separate questions. Do you want to be in training data? Do you want to be citable? Failure mode: treating them as one. That is what the copy-paste list does, and it is the whole bug.

3a. If you want citations, name the retrieval crawlers explicitly. A named group beats the wildcard group under RFC 9309, so a later blanket rule cannot sweep them up.

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Applebot
User-agent: Amzn-SearchBot
User-agent: meta-webindexer
User-agent: MistralAI-Index
User-agent: DuckAssistBot
Allow: /

User-agent: *
Disallow:

Failure mode: leaving it at a bare User-agent: * and calling it safe. It is, until someone adds a wildcard disallow for a staging path and takes the lot with it.

3b. If you want out of training but still retrievable, put the training group above the allow group. Amazonbot belongs here: Amazon says it "may be used to train Amazon AI models."

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: Amazonbot
User-agent: CCBot
User-agent: AI2Bot
Disallow: /

Failure mode: adding Applebot or OAI-SearchBot there because they look related. They are the retrieval side.

4. Leave Googlebot and bingbot out of it entirely. Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal," and it does not control AI Overviews. Google's AI guidance: "robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search." Failure mode: blocking Googlebot to escape AI Overviews. You leave Search. The levers are nosnippet, data-nosnippet and max-snippet.

5. Enforce at the server, not in the file. RFC 9309 says it plainly: "The Robots Exclusion Protocol is not a substitute for valid content security measures." Every vendor here publishes IP ranges (OpenAI's at openai.com/gptbot.json), and Perplexity's docs tell you to pair user-agent matching with IP verification in a WAF rule. Failure mode: believing robots.txt stopped a user-triggered fetch. Your logs will disagree.

6. Re-check after 24 hours, and after any CDN change. Caches vary: OpenAI and Perplexity about 24 hours, DuckDuckGo 72, Amazon up to 30 days. Failure mode: token rot. The blocklist ships Google-NotebookLM, which Google supports only until August 2026; the live token is Google-GeminiNotebook. It also blocks cohere-ai and cohere-training-data-crawler, two agents Cohere says it does not operate.

When is blocking actually right?

When the text is the product. For a metered publisher, a paid research service or a docs site behind a seat license, an assistant answering the question substitutes for the visit rather than causing it. Blocking is coherent there, and OpenAI's navigational-links carve-out is the shape of that deal: refuse the extraction, keep the link.

The test is what you sell. If you monetize the pageview, blocking has a price you can calculate. If you monetize being known, which is nearly every SaaS, agency and personal site, you pay that price for nothing.

How do you know it is working?

Group your access logs by user agent and count how many of the eight retrieval tokens appear. That number, out of 8, is your real exposure. Verify each hit against the vendor's published IP list, because the user-agent string is trivially spoofed. Then watch assistant referrals, instrumented properly rather than guessed at. A zero for a bot you meant to allow is a block, not a preference, and unlike an llms.txt nobody reads, it costs you.

What this does not prove: 30 tokens is what the vendors admit to. Bytespider and xAI publish no policy I could fetch, so there the server layer is your only control, and a written promise to honor robots.txt is not a mechanism. The count establishes something narrower. On the vendors' own documentation, blocking is not one decision but at least three, and the snippet makes all three for you.