AI crawler data · November 2026
ChatGPT-User
Operator: OpenAI · Purpose: Search & user fetch · Measured November 19, 2026 by AnswerIndex
ChatGPT-User is OpenAI’s user-action agent. It fetches a specific page when a ChatGPT user asks about it; because these visits are user-initiated, they are not bound by robots.txt in the way crawlers are, though the token still appears in robots.txt policies.
In our November 19, 2026 measurement of the Tranco top 10,000 websites, 1,245 of the 7,091 sites with a determinable robots policy block ChatGPT-User (17.6%), while 5,846 allow it (82.4%). That ranks ChatGPT-User #10 of the 14 AI agents measured, by blocked share.
ChatGPT-User in context
- Blocking ChatGPT-User is usually part of a blanket policy. 1,024 of the 1,245 sites that block ChatGPT-User (82.2%) also block every other search & user-fetch agents measured.
- But it is rarely the only target. Only 3 sites block ChatGPT-User while allowing all 13 other AI agents.
- Cross-group selectivity: 5 sites block ChatGPT-User yet allow every training & data-collection crawler — they discriminate by purpose, not against AI traffic in general.
- Among llms.txt publishers the picture differs. Of the 1,114 measured sites that both publish llms.txt and have a determinable robots policy, 23 block ChatGPT-User (2.1%), versus 17.6% across all determinable sites.
How to block or allow ChatGPT-User in robots.txt
robots.txt rules are matched by user-agent token. To block ChatGPT-User from an entire site:
User-agent: ChatGPT-User Disallow: /
To explicitly allow it (the default when no group mentions the token):
User-agent: ChatGPT-User Allow: /
AnswerIndex evaluates these rules per RFC 9309: an agent-specific group beats the * group, and a site whose robots.txt refuses with HTTP 401/403 counts as fully disallowed for every agent. Note that robots.txt is advisory — it records the policy a site declares, and user-action agents fetching at a person’s request may not be bound by it.
Source data
These figures come from the AI Crawler & llms.txt Adoption Index — November 2026, based on the Tranco top 10,000 measured on November 19, 2026. Compare all agents on the AI crawlers index, or download the full per-domain dataset: results-2026-11-19.csv.
Cite this: AnswerIndex, AI Crawler & llms.txt Adoption Index, November 2026. https://answerindex.org/reports/2026-11-ai-crawler-index/