AnswerIndex

Original measurement data on how the web treats AI crawlers.

AI crawler data · November 2026

Claude-User

Operator: Anthropic · Purpose: Search & user fetch · Measured November 19, 2026 by AnswerIndex

Claude-User is Anthropic’s user-action agent. It fetches a specific page when a Claude user asks about it.

In our November 19, 2026 measurement of the Tranco top 10,000 websites, 1,150 of the 7,091 sites with a determinable robots policy block Claude-User (16.2%), while 5,941 allow it (83.8%). That ranks Claude-User #12 of the 14 AI agents measured, by blocked share.

16.2%
of sites with a determinable robots policy block Claude-User
1,150
sites block Claude-User (of 7,091 determinable)
5,941
sites allow Claude-User (83.8%)
#12
of 14 AI agents, ranked by blocked share

Claude-User in context

How to block or allow Claude-User in robots.txt

robots.txt rules are matched by user-agent token. To block Claude-User from an entire site:

User-agent: Claude-User
Disallow: /

To explicitly allow it (the default when no group mentions the token):

User-agent: Claude-User
Allow: /

AnswerIndex evaluates these rules per RFC 9309: an agent-specific group beats the * group, and a site whose robots.txt refuses with HTTP 401/403 counts as fully disallowed for every agent. Note that robots.txt is advisory — it records the policy a site declares, and user-action agents fetching at a person’s request may not be bound by it.

Source data

These figures come from the AI Crawler & llms.txt Adoption Index — November 2026, based on the Tranco top 10,000 measured on November 19, 2026. Compare all agents on the AI crawlers index, or download the full per-domain dataset: results-2026-11-19.csv.

Cite this: AnswerIndex, AI Crawler & llms.txt Adoption Index, November 2026. https://answerindex.org/reports/2026-11-ai-crawler-index/