Glossary
ClaudeBot
Anthropic's web crawler, which fetches public content for use in Claude's training and live retrieval features.
By Teeming Chew, Founder Last updated
ClaudeBot is the primary Anthropic user-agent for crawling the public web. A secondary anthropic-ai agent is used for some specific retrieval flows.
How do you allow ClaudeBot?
User-agent: ClaudeBot followed by Allow: / in robots.txt. Most sites should also explicitly allow anthropic-ai and Claude-Web.
Does ClaudeBot respect robots.txt?
Yes. Anthropic has stated publicly that ClaudeBot honors robots.txt directives.
Part of the Cite Hustle GEO glossary: definitions for generative engine optimization and AI search. See how it fits the bigger picture in the GEO methodology.