CITEHUSTLE
Log in Get started

Anthropic · Training crawler

ClaudeBot

ClaudeBot is Anthropic's crawler that collects public web content which may be used to train and improve Claude. In February 2026 Anthropic split its crawling into three independently controllable bots — ClaudeBot (training), Claude-User (user fetches), and Claude-SearchBot (search indexing) — all of which honor robots.txt. Blocking ClaudeBot excludes your future content from Claude's training data without affecting Claude's search citations.

Last updated

User-agent token
ClaudeBot
Operator
Anthropic
Feeds
Claude model training
robots.txt
Unverified

How to control ClaudeBot in robots.txt

Edit the robots.txt file at the root of your domain (for example https://example.com/robots.txt), add one of the groups below, then save and re-deploy. Remember: a named User-agent: ClaudeBot group overrides your global User-agent: * rules, so repeat any private Disallow paths inside it.

Allow ClaudeBot (recommended for AI visibility)

# Welcome ClaudeBot, but keep private areas blocked.
# A named user-agent group overrides "User-agent: *", so repeat
# your own private Disallow rules inside this group.
User-agent: ClaudeBot
Allow: /
Disallow: /admin/
Disallow: /account/
Disallow: /cart/
Disallow: /checkout/

Block ClaudeBot

# Block ClaudeBot from the entire site.
User-agent: ClaudeBot
Disallow: /

FAQ

Does ClaudeBot obey robots.txt?

Yes. Anthropic states all three of its crawlers — ClaudeBot, Claude-User, and Claude-SearchBot — honor robots.txt and can be controlled separately.

If I block ClaudeBot, am I removed from Claude's search answers?

No. ClaudeBot only handles training. Claude's search citations are governed by Claude-SearchBot, which you control independently.

Is your site visible to AI crawlers?

Run a free AI-visibility audit to see which AI crawlers can reach your content and how often you get cited by ChatGPT, Perplexity, Claude, and Google AI Overviews.

Run a free audit

Part of the Cite Hustle AI crawler directory. For the full framework on AI search visibility, read the GEO methodology.