The AI crawler registry
This registry is the roster willaicite checks on every audit: 17 crawler tokens, what each one fetches, whether blocking it costs you citations or only model training, and the robots.txt lines that control it. The page renders the same data the audit engine scores with, so it cannot drift from what the tool actually tests.
Which crawlers decide whether you get cited?
Retrieval crawlers fetch pages to answer live queries or fill an answer engine’s search index. Blocking any one of them means that engine can never retrieve or cite your page, no matter how good the content is. 11 of the 17 tokens are retrieval-class.
| token | what it feeds | note |
|---|---|---|
| OAI-SearchBot | ChatGPT Search index | |
| ChatGPT-User | ChatGPT live browsing | |
| Claude-SearchBot | Claude search index | |
| Claude-User | Claude live browsing | |
| PerplexityBot | Perplexity index | |
| Perplexity-User | Perplexity live fetch (sends real referrals) | generally ignores robots.txt (user-initiated); a Disallow records intent but does not stop the fetch |
| Bingbot | Bing index (feeds Copilot and, residually, ChatGPT answers) | |
| Amazonbot | Alexa / Rufus answers | |
| DuckAssistBot | DuckDuckGo AI answers | |
| Applebot | Siri / Spotlight / Apple Intelligence retrieval | |
| MistralAI-User | Le Chat live fetch |
Which tokens only affect training?
Training tokens control whether your content ends up in model-training data. Blocking them is a legitimate, now-mainstream content policy and does not stop AI engines from citing your page today. Two of them (Google-Extended and Applebot-Extended) are pure opt-out tokens with no crawler behind them.
| token | what it controls | note |
|---|---|---|
| GPTBot | OpenAI model training | |
| ClaudeBot | Anthropic model training | |
| CCBot | Common Crawl (feeds many training sets) | |
| meta-externalagent | Meta AI training | |
| Google-Extended | Gemini training + grounding opt-out token (no crawler) | this token also gates Gemini grounding: blocking it stops Gemini from pulling and citing this content at answer time, not just from training on it |
| Applebot-Extended | Apple Intelligence training opt-out token (no crawler) |
Who actually blocks these crawlers?
The Reuters Institute measured it for news publishers: “48% of the most widely used news websites … were blocking OpenAI’s crawlers” by the end of 2023, across ten countries, while 24% blocked Google’s AI crawler (Fletcher, Reuters Institute, 2024). The spread is wide: 79% of sampled US news sites blocked OpenAI against 20% in Mexico and Poland, and 57% of legacy print outlets blocked it against 31% of digital-born ones. Those blocks are aimed at training crawlers; the risk willaicite flags is blocking a retrieval token by accident and vanishing from citations with them.
How do I allow or block a crawler?
robots.txt rules are per token. No rule means allowed; every crawler above is allowed by default. To block one, name it and disallow your paths:
# block OpenAI model training, keep ChatGPT citations
User-agent: GPTBot
Disallow: /
# block Gemini training and grounding
User-agent: Google-Extended
Disallow: /Two things robots.txt cannot tell you:
- User-initiated fetchers like Perplexity-User document that they generally ignore robots.txt, so a Disallow there records intent without stopping the fetch; real enforcement lives at the WAF or CDN.
- The reverse failure is silent: a firewall rule that returns 403 to AI crawlers overrides everything robots.txt promises, and no SEO tool walks that path. willaicite fetches your page with an AI crawler’s user agent and reports the difference; a clean result reads “UA differential: normal UA and GPTBot UA both got HTTP 200”.
Registry FAQ
Should I block GPTBot?
That is a content-policy decision, not a technical one. Blocking GPTBot keeps your pages out of OpenAI’s training data and does not remove you from ChatGPT’s citations; those run through OAI-SearchBot and ChatGPT-User. Many publishers block it deliberately. Just make sure the block stops at GPTBot and does not sweep up the retrieval tokens with it.
Why do AI crawlers get a 403 when robots.txt says allowed?
Because robots.txt and the firewall are separate layers. CDN bot-management rules (Cloudflare ships a one-click AI-bot block) act on the user agent at the edge, before your server or your robots.txt opinion is consulted. The audit’s crawler-access dimension flags this as its hardest failure: “a blocked retrieval crawler cannot fetch the page at all, so the engine behind it can never retrieve or cite this content.” Every SEO report still shows a clean crawl.
Does one Disallow line cover every AI bot?
No. Rules bind per token, so each crawler needs its own User-agent group unless you
use the * wildcard, which also binds ordinary search crawlers. And the
two pure opt-out tokens (Google-Extended, Applebot-Extended) have no crawler behind
them at all; they only register your preference.