The AI crawler list
Every major AI crawler, what it feeds, and the exact robots.txt lines to allow or block it. Updated as the bots change.
| User agent | Operator | What it feeds |
|---|---|---|
GPTBot |
OpenAI | training + retrieval for ChatGPT |
OAI-SearchBot |
OpenAI | ChatGPT search citations |
ChatGPT-User |
OpenAI | live browsing when users ask ChatGPT |
ClaudeBot |
Anthropic | Claude training + retrieval |
Claude-User |
Anthropic | live browsing when users ask Claude |
PerplexityBot |
Perplexity | Perplexity indexing + citations |
Perplexity-User |
Perplexity | live Perplexity answers |
Google-Extended |
Gemini training (does NOT affect Search/AI Overviews) | |
Applebot-Extended |
Apple | Apple Intelligence training |
meta-externalagent |
Meta | Meta AI training |
Bytespider |
ByteDance | Doubao/TikTok AI training |
CCBot |
Common Crawl | the open corpus many models train on |
Allow every AI crawler (get cited)
User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / User-agent: meta-externalagent Allow: / User-agent: Bytespider Allow: / User-agent: CCBot Allow: /Block every AI crawler
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-User Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Perplexity-User Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: meta-externalagent Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: /
Blocking training bots while wanting AI recommendations is the classic contradiction — check what your robots.txt actually says with our AI bot access checker.
This tool shows a snapshot. Asheard shows the movie.
Track whether ChatGPT, Gemini, Claude, Perplexity and Google's AI actually recommend you — daily, per city, against competitors.
Try Asheard free for 7 days