AnswerIndex

Original measurement data on how the web treats AI crawlers.

Sector report · November 2026

Do news sites block AI crawlers?

Measured November 19, 2026 · AnswerIndex

News publishers are the front line of the web’s fight over AI training data, and their robots.txt files show it. We identified 386 news outlets inside the Tranco top 10,000 from our November 2026 census and compared their policies with the web at large. The gap is wide on every measure: news sites block all 14 AI agents at roughly twice the all-site rate — and they are only half as likely to publish an llms.txt file.

57.0%
of news outlets with a determinable policy block CCBot — vs 25.8% of all sites
52.7%
block GPTBot — vs 24.9% of all sites
49.6%
average blocked share across the eight training crawlers — vs 23.9% across all sites
5.7%
of news outlets publish llms.txt — vs 11.23% of the Tranco top 10,000

News outlets vs all sites, agent by agent

Blocked share of sites with a determinable robots policy: 374 news outlets against 7,091 sites overall. Every percentage uses the same basis and the same November 19, 2026 snapshot as the main census.

User-agentPurposeNews outlets blockingNews mediaAll sitesRatio
CCBotTraining21357.0%25.8%2.2×
ClaudeBotTraining20053.5%24.1%2.2×
BytespiderTraining19852.9%25.7%2.1×
GPTBotTraining19752.7%24.9%2.1×
Applebot-ExtendedTraining17647.1%22.6%2.1×
PerplexityBotSearch / fetch17346.3%18.6%2.5×
Meta-ExternalAgentTraining17346.3%23.1%2.0×
Google-ExtendedTraining16443.9%22.7%1.9×
AmazonbotTraining16343.6%22.6%1.9×
ChatGPT-UserSearch / fetch14238.0%17.6%2.2×
Perplexity-UserSearch / fetch12132.4%16.6%2.0×
OAI-SearchBotSearch / fetch11931.8%15.9%2.0×
Claude-SearchBotSearch / fetch11831.6%15.9%2.0×
Claude-UserSearch / fetch11129.7%16.2%1.8×

Findings

The ten highest-ranked news outlets

Tranco rankOutletGPTBotCCBotllms.txtllms-full.txt
100t-online (t-online.de)BlockedBlockedNoNo
154The New York Times (nytimes.com)BlockedBlockedNoNo
178CNN (cnn.com)BlockedBlockedNoNo
199The Guardian (theguardian.com)AllowedBlockedNoNo
216BBC (bbc.com)BlockedBlockedNoNo
227Forbes (forbes.com)AllowedAllowedNoNo
243BBC (bbc.co.uk)BlockedBlockedNoNo
286Reuters (reuters.com)BlockedBlockedNoNo
331ESPN (espn.com)BlockedBlockedNoNo
357The Wall Street Journal (wsj.com)AllowedBlockedNoNo

Seven of the ten block GPTBot and nine block CCBot. The Guardian and the Wall Street Journal allow GPTBot but block CCBot; Forbes is the only one of the ten that allows both.

Methodology

The sample is the 386 domains in the November 2026 census (Tranco top 10,000, measured November 19, 2026) that we identified as news outlets. Candidates came from two routes: a hand-built seed list of major outlets — wire services, national newspapers, broadcast news organisations and digital-native news sites — and a scan of census domains for newsroom tokens (news, times, post, herald, tribune, journal, daily, gazette, press, chronicle, observer, telegraph, mirror, today, wire, bulletin, globe and similar). Every candidate was then judged by hand against one rule: the domain’s primary business is producing news or journalism, general or vertical (business, technology, sport, entertainment trade and science news included; news magazines included).

Excluded by that rule: aggregators, press-release distribution services, corporate and content-marketing sites, academic journals, government publications, pure scores / betting / data services, weather services, lifestyle-only magazines and opinion-only blogs. Both the outlet list and the per-domain results for the sample are published below, so every inclusion can be checked — and challenged.

All percentages use the census basis: agent blocked shares are computed over the 374 outlets with a determinable robots policy (robots.txt retrieved, or refused with HTTP 401/403, which counts as fully disallowed); llms.txt adoption is computed over all 386 outlets. Verdict rules are identical to the main report.

Limitations: a hand-built list involves judgement calls, and another researcher’s list would differ at the margin; the Tranco top 10,000 skews towards large, established outlets; policies are measured at the registered domain, so an outlet’s regional or section subdomains may differ; and all figures are a snapshot of November 19, 2026 — robots.txt files change without notice.

Data behind this report

Download news-media-results-2026-11.csv — the census rows for all 386 outlets.

Download news-media-list-2026-11.csv — the outlet list itself: domain, outlet name, and how each candidate was found (seed list or token scan).

Full census dataset: results-2026-11-19.csv · Mirrors: GitHub · Hugging Face

Tool: check any site’s AI-crawler policy live with the same rules as this census.

Get the next sector report by email

AnswerIndex re-measures the top of the web every quarter and publishes sector cuts like this one. Subscribe and we will email you once when the next edition lands.

One email per quarter. No spam. Unsubscribe anytime.

Cite this: AnswerIndex, Do News Sites Block AI Crawlers?, November 2026. https://answerindex.org/reports/2026-11-news-media-ai-crawlers/