AnswerIndex

Original measurement data on how the web treats AI crawlers.

Measurement release · AI Crawler & llms.txt Adoption Index

llms.txt adoption has roughly doubled in five months — and the web is blocking training, not AI

November 19, 2026 · AnswerIndex

On November 19, 2026, AnswerIndex measured the Tranco top 10,000 websites: their robots.txt policy toward 14 AI agents, and whether they publish an llms.txt or llms-full.txt file. 1,123 sites (11.23%) now publish llms.txt, roughly double the June 2026 baseline. At the same time, sites draw a sharp line between kinds of AI traffic: training crawlers are blocked by up to a quarter of sites, while search and user-fetch agents are blocked far less.

11.23%
of the Tranco top 10,000 publish llms.txt — 1,123 sites, up from 556 in June 2026
25.8%
of sites with a determinable robots policy block CCBot, the most-blocked AI agent
15.9%
block OAI-SearchBot and Claude-SearchBot, the least-blocked agents measured
10,000
domains measured; full per-domain dataset published below

Key findings

Who gets blocked: all 14 AI agents

Sites discriminate by purpose. Agents that collect content for model training and datasets cluster at the top of the blocking ranking; agents that power AI search results or fetch a page at a user’s request cluster at the bottom.

TRAINING & DATA-COLLECTION CRAWLERS CCBot 25.8% Bytespider 25.7% GPTBot 24.9% ClaudeBot 24.1% Meta-ExternalAgent 23.1% Google-Extended 22.7% Amazonbot 22.6% Applebot-Extended 22.6% SEARCH & USER-FETCH AGENTS PerplexityBot 18.6% ChatGPT-User 17.6% Perplexity-User 16.6% Claude-User 16.2% OAI-SearchBot 15.9% Claude-SearchBot 15.9%
Share of sites blocking each AI agent, out of the 7,091 top-10,000 sites with a determinable robots policy (robots.txt retrieved, or refused with HTTP 401/403, which counts as fully disallowed). Grouping follows each operator’s stated purpose for the agent. Measured November 19, 2026.

llms.txt adoption: June 2026 vs November 2026

In June 2026, Rankability measured llms.txt adoption across the same Tranco top-10,000 tier. Five months later, both files have roughly doubled:

FileJune 2026 (Rankability)November 19, 2026 (AnswerIndex)Change
llms.txt5561,123+567
llms-full.txt112222+110
Either file5581,126+568
Both files~110 (implied)219—

Today, 1,123 sites (11.23% of the top 10,000) publish llms.txt and 222 (2.22%) publish llms-full.txt; 1,126 publish at least one of the two. The comparison is directional, not exact: the Tranco list changes daily, so the two samples are not identical domain sets, and counting rules may differ slightly (this crawl follows redirects and rejects HTML soft-404 bodies).

Full results: AI crawler access in robots.txt

Of the 10,000 domains measured, robots.txt was retrieved for 6,397 (64.0%); 694 (6.9%) refused with HTTP 401/403 and are treated as fully disallowed; for 2,909 (29.1%) robots.txt was missing or unreachable. Blocked percentages below are computed over the 7,091 sites with a determinable policy.

User-agentBlockedBlocked %Allowed
GPTBot1,76424.9%5,327
OAI-SearchBot1,12715.9%5,964
ChatGPT-User1,24517.6%5,846
ClaudeBot1,71124.1%5,380
Claude-SearchBot1,13115.9%5,960
Claude-User1,15016.2%5,941
PerplexityBot1,32118.6%5,770
Perplexity-User1,17516.6%5,916
Google-Extended1,61122.7%5,480
CCBot1,82825.8%5,263
Bytespider1,82125.7%5,270
Meta-ExternalAgent1,63723.1%5,454
Amazonbot1,60222.6%5,489
Applebot-Extended1,60122.6%5,490

Methodology & limitations

How the crawl worked

On November 19, 2026 we took the first 10,000 domains of the Tranco top-1M list (snapshot downloaded the same day) and probed each domain for three files: /robots.txt, /llms.txt, and /llms-full.txt. An llms file counts as present only when the probe returns HTTP 200 with a non-HTML body — an HTML response is treated as a soft-404 and counted as absent; redirects are followed. Each retrieved robots.txt was evaluated for 14 AI user-agents using RFC 9309 group selection: an agent-specific group beats the * group, the longest matching rule for / wins, and Allow wins ties. A robots.txt refused with HTTP 401/403 is treated as fully disallowed for every agent; other non-200 responses and transport failures are recorded as no determinable policy.

Limitations — read these before citing

Dataset

The complete per-domain dataset behind this report — all 10,000 domains, robots.txt status, the verdict for each of the 14 agents, and llms.txt / llms-full.txt presence — is published as a CSV:

Download results-2026-11-19.csv

Cite this: AnswerIndex, AI Crawler & llms.txt Adoption Index, November 2026. https://answerindex.org/reports/2026-11-ai-crawler-index/