AnswerIndex

Original measurement data on how the web treats AI crawlers.

About AnswerIndex

AnswerIndex is an independent measurement publication. We run our own crawls and measurements of how the web treats AI crawlers — and how AI search engines answer — and we publish the data itself, not commentary about other people’s data.

The first release, the AI Crawler & llms.txt Adoption Index (November 2026), measured the Tranco top 10,000 websites: their robots.txt policy toward 14 AI agents and their adoption of llms.txt. The full per-domain dataset is published with the report.

Data series

Methodology principles