AI crawler data · November 2026
AI crawlers: who gets blocked, and how often
Measured November 19, 2026 · AnswerIndex
AnswerIndex measured how the Tranco top 10,000 websites treat 14 AI agents in robots.txt. Of the 10,000 domains, 7,091 had a determinable robots policy (robots.txt retrieved, or refused with HTTP 401/403, which counts as fully disallowed); every percentage on this page is a share of those 7,091 sites. Grouping follows each operator’s stated purpose for the agent.
All 14 AI agents, ranked by blocked share
| User-agent | Operator | Purpose | Blocked | Blocked % | Allowed |
|---|---|---|---|---|---|
| CCBot | Common Crawl | Training & data collection | 1,828 | 25.8% | 5,263 |
| Bytespider | ByteDance | Training & data collection | 1,821 | 25.7% | 5,270 |
| GPTBot | OpenAI | Training & data collection | 1,764 | 24.9% | 5,327 |
| ClaudeBot | Anthropic | Training & data collection | 1,711 | 24.1% | 5,380 |
| Meta-ExternalAgent | Meta | Training & data collection | 1,637 | 23.1% | 5,454 |
| Google-Extended | Training & data collection | 1,611 | 22.7% | 5,480 | |
| Amazonbot | Amazon | Training & data collection | 1,602 | 22.6% | 5,489 |
| Applebot-Extended | Apple | Training & data collection | 1,601 | 22.6% | 5,490 |
| PerplexityBot | Perplexity | Search & user fetch | 1,321 | 18.6% | 5,770 |
| ChatGPT-User | OpenAI | Search & user fetch | 1,245 | 17.6% | 5,846 |
| Perplexity-User | Perplexity | Search & user fetch | 1,175 | 16.6% | 5,916 |
| Claude-User | Anthropic | Search & user fetch | 1,150 | 16.2% | 5,941 |
| Claude-SearchBot | Anthropic | Search & user fetch | 1,131 | 15.9% | 5,960 |
| OAI-SearchBot | OpenAI | Search & user fetch | 1,127 | 15.9% | 5,964 |
Training and data-collection crawlers cluster at the top of the ranking; search and user-fetch agents cluster at the bottom. Open any agent’s page for its numbers in context and the robots.txt rules that control it.
Full report: AI Crawler & llms.txt Adoption Index — November 2026 · Dataset: results-2026-11-19.csv