AnswerIndex

Original measurement data on how the web treats AI crawlers.

AI crawler data · November 2026

OAI-SearchBot

Operator: OpenAI · Purpose: Search & user fetch · Measured November 19, 2026 by AnswerIndex

OAI-SearchBot is OpenAI’s search crawler. It surfaces websites in ChatGPT search results; OpenAI states it is not used to crawl content for training its foundation models.

In our November 19, 2026 measurement of the Tranco top 10,000 websites, 1,127 of the 7,091 sites with a determinable robots policy block OAI-SearchBot (15.9%), while 5,964 allow it (84.1%). That ranks OAI-SearchBot #14 of the 14 AI agents measured, by blocked share.

15.9%
of sites with a determinable robots policy block OAI-SearchBot
1,127
sites block OAI-SearchBot (of 7,091 determinable)
5,964
sites allow OAI-SearchBot (84.1%)
#14
of 14 AI agents, ranked by blocked share

OAI-SearchBot in context

How to block or allow OAI-SearchBot in robots.txt

robots.txt rules are matched by user-agent token. To block OAI-SearchBot from an entire site:

User-agent: OAI-SearchBot
Disallow: /

To explicitly allow it (the default when no group mentions the token):

User-agent: OAI-SearchBot
Allow: /

AnswerIndex evaluates these rules per RFC 9309: an agent-specific group beats the * group, and a site whose robots.txt refuses with HTTP 401/403 counts as fully disallowed for every agent. Note that robots.txt is advisory — it records the policy a site declares, and user-action agents fetching at a person’s request may not be bound by it.

Source data

These figures come from the AI Crawler & llms.txt Adoption Index — November 2026, based on the Tranco top 10,000 measured on November 19, 2026. Compare all agents on the AI crawlers index, or download the full per-domain dataset: results-2026-11-19.csv.

Cite this: AnswerIndex, AI Crawler & llms.txt Adoption Index, November 2026. https://answerindex.org/reports/2026-11-ai-crawler-index/