Sector report · November 2026
Do news sites block AI crawlers?
Measured November 19, 2026 · AnswerIndex
News publishers are the front line of the web’s fight over AI training data, and their robots.txt files show it. We identified 386 news outlets inside the Tranco top 10,000 from our November 2026 census and compared their policies with the web at large. The gap is wide on every measure: news sites block all 14 AI agents at roughly twice the all-site rate — and they are only half as likely to publish an llms.txt file.
News outlets vs all sites, agent by agent
Blocked share of sites with a determinable robots policy: 374 news outlets against 7,091 sites overall. Every percentage uses the same basis and the same November 19, 2026 snapshot as the main census.
| User-agent | Purpose | News outlets blocking | News media | All sites | Ratio |
|---|---|---|---|---|---|
| CCBot | Training | 213 | 57.0% | 25.8% | 2.2× |
| ClaudeBot | Training | 200 | 53.5% | 24.1% | 2.2× |
| Bytespider | Training | 198 | 52.9% | 25.7% | 2.1× |
| GPTBot | Training | 197 | 52.7% | 24.9% | 2.1× |
| Applebot-Extended | Training | 176 | 47.1% | 22.6% | 2.1× |
| PerplexityBot | Search / fetch | 173 | 46.3% | 18.6% | 2.5× |
| Meta-ExternalAgent | Training | 173 | 46.3% | 23.1% | 2.0× |
| Google-Extended | Training | 164 | 43.9% | 22.7% | 1.9× |
| Amazonbot | Training | 163 | 43.6% | 22.6% | 1.9× |
| ChatGPT-User | Search / fetch | 142 | 38.0% | 17.6% | 2.2× |
| Perplexity-User | Search / fetch | 121 | 32.4% | 16.6% | 2.0× |
| OAI-SearchBot | Search / fetch | 119 | 31.8% | 15.9% | 2.0× |
| Claude-SearchBot | Search / fetch | 118 | 31.6% | 15.9% | 2.0× |
| Claude-User | Search / fetch | 111 | 29.7% | 16.2% | 1.8× |
Findings
- Blocking is the norm in news, the exception elsewhere. 246 of the 374 determinable outlets (65.8%) block at least one AI agent, and only 128 (34.2%) block none at all — across the whole top 10,000, 4,966 of 7,091 sites (70.0%) block none.
- Training crawlers take the heaviest fire. All eight training and data-collection crawlers are blocked by 43.6–57.0% of news outlets, and 101 outlets (27.0%) block all eight at once, against 18.3% of sites overall.
- Answer engines are treated as rivals, not referrers. The widest gap of any agent is PerplexityBot: blocked by 46.3% of news outlets, 2.5× its all-site rate. Even user-fetch agents that retrieve a page at a reader’s request — ChatGPT-User (38.0%), Perplexity-User (32.4%), Claude-User (29.7%) — are blocked at about twice the web-wide rate.
- The llms.txt paradox. The sector that blocks AI crawlers hardest is the slowest to adopt the file meant for them: 22 of 386 outlets (5.7%) publish llms.txt, against 11.23% of the top 10,000, and just 2 (0.5%) publish llms-full.txt, against 2.22%. None of the ten highest-ranked outlets below publishes either file.
The ten highest-ranked news outlets
| Tranco rank | Outlet | GPTBot | CCBot | llms.txt | llms-full.txt |
|---|---|---|---|---|---|
| 100 | t-online (t-online.de) | Blocked | Blocked | No | No |
| 154 | The New York Times (nytimes.com) | Blocked | Blocked | No | No |
| 178 | CNN (cnn.com) | Blocked | Blocked | No | No |
| 199 | The Guardian (theguardian.com) | Allowed | Blocked | No | No |
| 216 | BBC (bbc.com) | Blocked | Blocked | No | No |
| 227 | Forbes (forbes.com) | Allowed | Allowed | No | No |
| 243 | BBC (bbc.co.uk) | Blocked | Blocked | No | No |
| 286 | Reuters (reuters.com) | Blocked | Blocked | No | No |
| 331 | ESPN (espn.com) | Blocked | Blocked | No | No |
| 357 | The Wall Street Journal (wsj.com) | Allowed | Blocked | No | No |
Seven of the ten block GPTBot and nine block CCBot. The Guardian and the Wall Street Journal allow GPTBot but block CCBot; Forbes is the only one of the ten that allows both.
Methodology
The sample is the 386 domains in the November 2026 census (Tranco top 10,000, measured November 19, 2026) that we identified as news outlets. Candidates came from two routes: a hand-built seed list of major outlets — wire services, national newspapers, broadcast news organisations and digital-native news sites — and a scan of census domains for newsroom tokens (news, times, post, herald, tribune, journal, daily, gazette, press, chronicle, observer, telegraph, mirror, today, wire, bulletin, globe and similar). Every candidate was then judged by hand against one rule: the domain’s primary business is producing news or journalism, general or vertical (business, technology, sport, entertainment trade and science news included; news magazines included).
Excluded by that rule: aggregators, press-release distribution services, corporate and content-marketing sites, academic journals, government publications, pure scores / betting / data services, weather services, lifestyle-only magazines and opinion-only blogs. Both the outlet list and the per-domain results for the sample are published below, so every inclusion can be checked — and challenged.
All percentages use the census basis: agent blocked shares are computed over the 374 outlets with a determinable robots policy (robots.txt retrieved, or refused with HTTP 401/403, which counts as fully disallowed); llms.txt adoption is computed over all 386 outlets. Verdict rules are identical to the main report.
Limitations: a hand-built list involves judgement calls, and another researcher’s list would differ at the margin; the Tranco top 10,000 skews towards large, established outlets; policies are measured at the registered domain, so an outlet’s regional or section subdomains may differ; and all figures are a snapshot of November 19, 2026 — robots.txt files change without notice.
Data behind this report
Download news-media-results-2026-11.csv — the census rows for all 386 outlets.
Download news-media-list-2026-11.csv — the outlet list itself: domain, outlet name, and how each candidate was found (seed list or token scan).
Full census dataset: results-2026-11-19.csv · Mirrors: GitHub · Hugging Face
Tool: check any site’s AI-crawler policy live with the same rules as this census.
Get the next sector report by email
AnswerIndex re-measures the top of the web every quarter and publishes sector cuts like this one. Subscribe and we will email you once when the next edition lands.
One email per quarter. No spam. Unsubscribe anytime.
Cite this: AnswerIndex, Do News Sites Block AI Crawlers?, November 2026. https://answerindex.org/reports/2026-11-news-media-ai-crawlers/