AI crawler data · November 2026
Perplexity-User
Operator: Perplexity · Purpose: Search & user fetch · Measured November 19, 2026 by AnswerIndex
Perplexity-User is Perplexity’s user-action agent. It fetches a specific page when a Perplexity user asks about it.
In our November 19, 2026 measurement of the Tranco top 10,000 websites, 1,175 of the 7,091 sites with a determinable robots policy block Perplexity-User (16.6%), while 5,916 allow it (83.4%). That ranks Perplexity-User #11 of the 14 AI agents measured, by blocked share.
Perplexity-User in context
- Blocking Perplexity-User is usually part of a blanket policy. 1,024 of the 1,175 sites that block Perplexity-User (87.1%) also block every other search & user-fetch agents measured.
- But it is rarely the only target. Only 2 sites block Perplexity-User while allowing all 13 other AI agents.
- Cross-group selectivity: 8 sites block Perplexity-User yet allow every training & data-collection crawler — they discriminate by purpose, not against AI traffic in general.
- Among llms.txt publishers the picture differs. Of the 1,114 measured sites that both publish llms.txt and have a determinable robots policy, 25 block Perplexity-User (2.2%), versus 16.6% across all determinable sites.
How to block or allow Perplexity-User in robots.txt
robots.txt rules are matched by user-agent token. To block Perplexity-User from an entire site:
User-agent: Perplexity-User Disallow: /
To explicitly allow it (the default when no group mentions the token):
User-agent: Perplexity-User Allow: /
AnswerIndex evaluates these rules per RFC 9309: an agent-specific group beats the * group, and a site whose robots.txt refuses with HTTP 401/403 counts as fully disallowed for every agent. Note that robots.txt is advisory — it records the policy a site declares, and user-action agents fetching at a person’s request may not be bound by it.
Source data
These figures come from the AI Crawler & llms.txt Adoption Index — November 2026, based on the Tranco top 10,000 measured on November 19, 2026. Compare all agents on the AI crawlers index, or download the full per-domain dataset: results-2026-11-19.csv.
Cite this: AnswerIndex, AI Crawler & llms.txt Adoption Index, November 2026. https://answerindex.org/reports/2026-11-ai-crawler-index/