Reference
AI crawlers, user agents & robots.txt rules.
AI crawlers are bots that fetch web pages to train AI models, build AI search indexes, or answer a user's question in real time. Allowing the right ones is how your content becomes eligible to be cited by ChatGPT, Claude, Perplexity, Google AI Overviews, and Microsoft Copilot. Below is every major AI crawler — its user-agent token, what it feeds, whether it honors robots.txt, and copy-paste rules to allow or block it.
For how crawler access fits the bigger picture, read the GEO methodology, or generate a complete file with the robots.txt builder.
Honors robots.txt
Partial / disputed
Unverified