AnswerIndex

Original measurement data on how the web treats AI crawlers.

AI crawler data · November 2026

Perplexity-User

Operator: Perplexity · Purpose: Search & user fetch · Measured November 19, 2026 by AnswerIndex

Perplexity-User is Perplexity’s user-action agent. It fetches a specific page when a Perplexity user asks about it.

In our November 19, 2026 measurement of the Tranco top 10,000 websites, 1,175 of the 7,091 sites with a determinable robots policy block Perplexity-User (16.6%), while 5,916 allow it (83.4%). That ranks Perplexity-User #11 of the 14 AI agents measured, by blocked share.

16.6%
of sites with a determinable robots policy block Perplexity-User
1,175
sites block Perplexity-User (of 7,091 determinable)
5,916
sites allow Perplexity-User (83.4%)
#11
of 14 AI agents, ranked by blocked share

Perplexity-User in context

How to block or allow Perplexity-User in robots.txt

robots.txt rules are matched by user-agent token. To block Perplexity-User from an entire site:

User-agent: Perplexity-User
Disallow: /

To explicitly allow it (the default when no group mentions the token):

User-agent: Perplexity-User
Allow: /

AnswerIndex evaluates these rules per RFC 9309: an agent-specific group beats the * group, and a site whose robots.txt refuses with HTTP 401/403 counts as fully disallowed for every agent. Note that robots.txt is advisory — it records the policy a site declares, and user-action agents fetching at a person’s request may not be bound by it.

Source data

These figures come from the AI Crawler & llms.txt Adoption Index — November 2026, based on the Tranco top 10,000 measured on November 19, 2026. Compare all agents on the AI crawlers index, or download the full per-domain dataset: results-2026-11-19.csv.

Cite this: AnswerIndex, AI Crawler & llms.txt Adoption Index, November 2026. https://answerindex.org/reports/2026-11-ai-crawler-index/