AI crawler data · November 2026
Claude-User
Operator: Anthropic · Purpose: Search & user fetch · Measured November 19, 2026 by AnswerIndex
Claude-User is Anthropic’s user-action agent. It fetches a specific page when a Claude user asks about it.
In our November 19, 2026 measurement of the Tranco top 10,000 websites, 1,150 of the 7,091 sites with a determinable robots policy block Claude-User (16.2%), while 5,941 allow it (83.8%). That ranks Claude-User #12 of the 14 AI agents measured, by blocked share.
Claude-User in context
- Blocking Claude-User is usually part of a blanket policy. 1,024 of the 1,150 sites that block Claude-User (89.0%) also block every other search & user-fetch agents measured.
- But it is rarely the only target. Only 0 sites block Claude-User while allowing all 13 other AI agents.
- Cross-group selectivity: 2 sites block Claude-User yet allow every training & data-collection crawler — they discriminate by purpose, not against AI traffic in general.
- Among llms.txt publishers the picture differs. Of the 1,114 measured sites that both publish llms.txt and have a determinable robots policy, 22 block Claude-User (2.0%), versus 16.2% across all determinable sites.
How to block or allow Claude-User in robots.txt
robots.txt rules are matched by user-agent token. To block Claude-User from an entire site:
User-agent: Claude-User Disallow: /
To explicitly allow it (the default when no group mentions the token):
User-agent: Claude-User Allow: /
AnswerIndex evaluates these rules per RFC 9309: an agent-specific group beats the * group, and a site whose robots.txt refuses with HTTP 401/403 counts as fully disallowed for every agent. Note that robots.txt is advisory — it records the policy a site declares, and user-action agents fetching at a person’s request may not be bound by it.
Source data
These figures come from the AI Crawler & llms.txt Adoption Index — November 2026, based on the Tranco top 10,000 measured on November 19, 2026. Compare all agents on the AI crawlers index, or download the full per-domain dataset: results-2026-11-19.csv.
Cite this: AnswerIndex, AI Crawler & llms.txt Adoption Index, November 2026. https://answerindex.org/reports/2026-11-ai-crawler-index/