AnswerIndex

Original measurement data on how the web treats AI crawlers.

AnswerIndex · Measurement, not commentary

How the web treats AI crawlers, measured

AnswerIndex is an independent measurement publication. We crawl the top of the web ourselves and publish the numbers — how many sites block each AI crawler in robots.txt, how fast llms.txt is spreading — together with the full dataset behind every figure, so anyone can check the arithmetic.

Latest report: AI Crawler & llms.txt Adoption Index — November 2026

Measured November 19, 2026 · Tranco top 10,000 websites

On November 19, 2026 we measured the robots.txt policy of the Tranco top 10,000 toward 14 AI agents, and whether each site publishes an llms.txt or llms-full.txt file. Two findings stand out:

11.23%
of the Tranco top 10,000 publish llms.txt — 1,123 sites, roughly double the June 2026 baseline of 556
25.8%
of sites with a determinable robots policy block CCBot, the most-blocked AI agent
15.9%
block OAI-SearchBot and Claude-SearchBot, the least-blocked agents measured
10,000
domains measured; the full per-domain dataset is published, free to use with attribution

Read the full November 2026 report — charts, the complete 14-agent table, methodology and limitations.

Reports

AI crawler data

Per-agent pages with the November 2026 numbers, what each agent is, and the robots.txt rules that block or allow it:

Check your own site

See how any domain treats the same 14 AI crawlers, checked live against its current robots.txt and llms.txt files: the AI crawler checker. Verdicts use exactly the rules of the census and are shown next to the top-10,000 blocked rates.

Dataset

The complete per-domain dataset behind the November 2026 report — all 10,000 domains, robots.txt status, the verdict for each of the 14 agents, and llms.txt / llms-full.txt presence — is published as a CSV, free to use with attribution:

Download results-2026-11-19.csv

Dataset mirrors: GitHub · Hugging Face

Cite this: AnswerIndex, AI Crawler & llms.txt Adoption Index, November 2026. https://answerindex.org/reports/2026-11-ai-crawler-index/

Get the next census by email

AnswerIndex re-measures the top of the web every quarter. Leave your email and we will send one message when the next edition lands — the new numbers, what changed, and the fresh dataset.

One email per quarter. No spam. Unsubscribe anytime.

Methodology

Primary data only: every number comes from a measurement we ran ourselves, with the crawl date and sample stated, known limitations published alongside the findings, and the underlying dataset released in full. The report’s methodology section documents the November 2026 crawl in detail; About AnswerIndex states the principles behind the series.