Reference
AI crawlers, user agents, and robots.txt rules.
AI crawlers fetch pages for model training, search indexes, or user-triggered answers. This directory separates those jobs and records each crawler's user-agent token, robots behavior, category, and published operator guidance. Allowing a crawler can make retrieval possible, but crawler access alone does not establish indexing, ranking, mention, or citation.
For how crawler access fits the bigger picture, read the GEO methodology, or generate a complete file with the robots.txt builder.
Honors robots.txt
Partial / disputed
Unverified
Which AI crawlers are in this directory?
OpenAI
Anthropic
Microsoft
Perplexity
Amazon
Apple
ByteDance
Common Crawl
Meta
Which of these crawlers can actually reach your site?
Run the free AI crawler access checker: it reads your live robots.txt and raw HTML, then reports Accessible, Limited, or Blocked for ChatGPT, Claude, Perplexity, Gemini, and Copilot. It does not measure citations.
Check AI crawler access