CITEHUSTLE
Free audit
Anthropic · Training crawler

ClaudeBot

ClaudeBot is Anthropic's crawler that collects public web content which may be used to train and improve Claude. In February 2026 Anthropic split its crawling into three independently controllable bots: ClaudeBot (training), Claude-User (user fetches), and Claude-SearchBot (search indexing), all of which honor robots.txt. Blocking ClaudeBot excludes your future content from Claude's training data without affecting Claude's search citations.

Last updated

What does ClaudeBot do?

User-agent token
ClaudeBot
Operator
Anthropic
Feeds
Claude model training
robots.txt
Unverified

How do you control ClaudeBot in robots.txt?

Edit the robots.txt file at the root of your domain (for example https://example.com/robots.txt), add one of the groups below, then save and re-deploy. Remember: a named User-agent: ClaudeBot group overrides your global User-agent: * rules, so repeat any private Disallow paths inside it.

Allow ClaudeBot (recommended for AI visibility)

# Welcome ClaudeBot, but keep private areas blocked.
# A named user-agent group overrides "User-agent: *", so repeat
# your own private Disallow rules inside this group.
User-agent: ClaudeBot
Allow: /
Disallow: /admin/
Disallow: /account/
Disallow: /cart/
Disallow: /checkout/

Block ClaudeBot

# Block ClaudeBot from the entire site.
User-agent: ClaudeBot
Disallow: /

FAQ

Does ClaudeBot obey robots.txt?

Yes. Anthropic states all three of its crawlers, ClaudeBot, Claude-User, and Claude-SearchBot, honor robots.txt and can be controlled separately.

If I block ClaudeBot, am I removed from Claude's search answers?

No. ClaudeBot only handles training. Claude's search citations are governed by Claude-SearchBot, which you control independently.

Is your site visible to AI crawlers?

Run a free technical access audit to inspect published crawler controls and raw HTML delivery. It does not measure whether an engine indexed or cited the site.

Run a free audit

Part of the Cite Hustle AI crawler directory. For the full framework on AI search visibility, read the GEO methodology.