AI crawler data · November 2026
CCBot
Operator: Common Crawl · Purpose: Training & data collection · Measured November 19, 2026 by AnswerIndex
CCBot is the crawler of Common Crawl, the non-profit that publishes open web crawl archives. Those archives are widely reused as AI training data, which is why CCBot is grouped with the training crawlers.
In our November 19, 2026 measurement of the Tranco top 10,000 websites, 1,828 of the 7,091 sites with a determinable robots policy block CCBot (25.8%), while 5,263 allow it (74.2%). That ranks CCBot #1 of the 14 AI agents measured, by blocked share.
CCBot in context
- Blocking CCBot is usually part of a blanket policy. 1,296 of the 1,828 sites that block CCBot (70.9%) also block every other training & data-collection crawlers measured.
- But it is rarely the only target. Only 24 sites block CCBot while allowing all 13 other AI agents.
- Cross-group selectivity: 477 sites block CCBot yet allow every search & user-fetch agent — they discriminate by purpose, not against AI traffic in general.
- Among llms.txt publishers the picture differs. Of the 1,114 measured sites that both publish llms.txt and have a determinable robots policy, 107 block CCBot (9.6%), versus 25.8% across all determinable sites.
How to block or allow CCBot in robots.txt
robots.txt rules are matched by user-agent token. To block CCBot from an entire site:
User-agent: CCBot Disallow: /
To explicitly allow it (the default when no group mentions the token):
User-agent: CCBot Allow: /
AnswerIndex evaluates these rules per RFC 9309: an agent-specific group beats the * group, and a site whose robots.txt refuses with HTTP 401/403 counts as fully disallowed for every agent. Note that robots.txt is advisory — it records the policy a site declares, and user-action agents fetching at a person’s request may not be bound by it.
Source data
These figures come from the AI Crawler & llms.txt Adoption Index — November 2026, based on the Tranco top 10,000 measured on November 19, 2026. Compare all agents on the AI crawlers index, or download the full per-domain dataset: results-2026-11-19.csv.
Cite this: AnswerIndex, AI Crawler & llms.txt Adoption Index, November 2026. https://answerindex.org/reports/2026-11-ai-crawler-index/