Tool · Live check
AI crawler checker
Enter any domain and this checker fetches its robots.txt live, evaluates it for the same 14 AI crawlers as the AnswerIndex census, and checks whether it publishes llms.txt or llms-full.txt — every verdict shown next to the top-10,000 average.
Get the quarterly census by email
The verdicts above are live; the comparison rates come from the quarterly census of the Tranco top 10,000. Subscribe to get the next edition — updated blocked rates for all 14 crawlers — by email when it lands.
One email per quarter. No spam. Unsubscribe anytime.
How verdicts are decided
The rules are identical to the November 2026 census: RFC 9309 group selection — an agent-specific group beats the * group, the longest rule matching / wins, and Allow wins ties. A robots.txt that refuses with HTTP 401/403 counts as fully disallowed for every agent. If no robots.txt can be retrieved, every crawler is allowed by default. An llms file counts as present only on HTTP 200 with a non-HTML body, so soft-404 pages do not count. Results are fetched live and cached for one hour; the comparison column is the blocked share among the 7,091 Tranco top-10,000 sites with a determinable robots policy, measured November 19, 2026.
Per-agent background and the full ranking: AI crawlers compared.