AnswerIndex · Measurement, not commentary
How the web treats AI crawlers, measured
AnswerIndex is an independent measurement publication. We crawl the top of the web ourselves and publish the numbers — how many sites block each AI crawler in robots.txt, how fast llms.txt is spreading — together with the full dataset behind every figure, so anyone can check the arithmetic.
Latest report: AI Crawler & llms.txt Adoption Index — November 2026
Measured November 19, 2026 · Tranco top 10,000 websites
On November 19, 2026 we measured the robots.txt policy of the Tranco top 10,000 toward 14 AI agents, and whether each site publishes an llms.txt or llms-full.txt file. Two findings stand out:
- llms.txt roughly doubled in about five months — 1,123 sites (11.23%) now, against 556 in June 2026; llms-full.txt doubled too, from 112 to 222 sites (2.22%).
- The web is not blocking AI — it is blocking training. Training and data-collection crawlers are blocked by 22.6–25.8% of sites with a determinable policy; search and user-fetch agents by 18.6% or less.
- Blocking is still a minority position. Even the most-blocked agent, CCBot, is allowed by 74.2% of sites with a determinable policy.
Read the full November 2026 report — charts, the complete 14-agent table, methodology and limitations.
Reports
- AI Crawler & llms.txt Adoption Index — November 2026 · First edition. Next edition: Q1 2027.
- Do News Sites Block AI Crawlers? — November 2026 · Sector report: news outlets block AI crawlers at about twice the web-wide rate (CCBot 57.0%), and publish llms.txt at half the average rate (5.7%).
AI crawler data
Per-agent pages with the November 2026 numbers, what each agent is, and the robots.txt rules that block or allow it:
- Most blocked: CCBot (25.8%), Bytespider (25.7%), GPTBot (24.9%)
- Least blocked: OAI-SearchBot (15.9%), Claude-SearchBot (15.9%), Claude-User (16.2%)
- All 14 AI crawlers compared
Check your own site
See how any domain treats the same 14 AI crawlers, checked live against its current robots.txt and llms.txt files: the AI crawler checker. Verdicts use exactly the rules of the census and are shown next to the top-10,000 blocked rates.
Dataset
The complete per-domain dataset behind the November 2026 report — all 10,000 domains, robots.txt status, the verdict for each of the 14 agents, and llms.txt / llms-full.txt presence — is published as a CSV, free to use with attribution:
Download results-2026-11-19.csv
Dataset mirrors: GitHub · Hugging Face
Cite this: AnswerIndex, AI Crawler & llms.txt Adoption Index, November 2026. https://answerindex.org/reports/2026-11-ai-crawler-index/
Get the next census by email
AnswerIndex re-measures the top of the web every quarter. Leave your email and we will send one message when the next edition lands — the new numbers, what changed, and the fresh dataset.
One email per quarter. No spam. Unsubscribe anytime.
Methodology
Primary data only: every number comes from a measurement we ran ourselves, with the crawl date and sample stated, known limitations published alongside the findings, and the underlying dataset released in full. The report’s methodology section documents the November 2026 crawl in detail; About AnswerIndex states the principles behind the series.