AnswerIndex

Original measurement data on how the web treats AI crawlers.

AI crawler data · November 2026

AI crawlers: who gets blocked, and how often

Measured November 19, 2026 · AnswerIndex

AnswerIndex measured how the Tranco top 10,000 websites treat 14 AI agents in robots.txt. Of the 10,000 domains, 7,091 had a determinable robots policy (robots.txt retrieved, or refused with HTTP 401/403, which counts as fully disallowed); every percentage on this page is a share of those 7,091 sites. Grouping follows each operator’s stated purpose for the agent.

2,125
of 7,091 sites block at least one of the 14 AI agents
4,966
block none of them — no AI-specific robots restriction at all
1,296
block all eight training & data-collection crawlers
1,024
block all six search & user-fetch agents

All 14 AI agents, ranked by blocked share

User-agentOperatorPurposeBlockedBlocked %Allowed
CCBotCommon CrawlTraining & data collection1,82825.8%5,263
BytespiderByteDanceTraining & data collection1,82125.7%5,270
GPTBotOpenAITraining & data collection1,76424.9%5,327
ClaudeBotAnthropicTraining & data collection1,71124.1%5,380
Meta-ExternalAgentMetaTraining & data collection1,63723.1%5,454
Google-ExtendedGoogleTraining & data collection1,61122.7%5,480
AmazonbotAmazonTraining & data collection1,60222.6%5,489
Applebot-ExtendedAppleTraining & data collection1,60122.6%5,490
PerplexityBotPerplexitySearch & user fetch1,32118.6%5,770
ChatGPT-UserOpenAISearch & user fetch1,24517.6%5,846
Perplexity-UserPerplexitySearch & user fetch1,17516.6%5,916
Claude-UserAnthropicSearch & user fetch1,15016.2%5,941
Claude-SearchBotAnthropicSearch & user fetch1,13115.9%5,960
OAI-SearchBotOpenAISearch & user fetch1,12715.9%5,964

Training and data-collection crawlers cluster at the top of the ranking; search and user-fetch agents cluster at the bottom. Open any agent’s page for its numbers in context and the robots.txt rules that control it.

Full report: AI Crawler & llms.txt Adoption Index — November 2026 · Dataset: results-2026-11-19.csv