Anthropic · Search indexer
Claude-SearchBot
Claude-SearchBot crawls and indexes content to improve the quality and relevance of Claude's search results. It is the Anthropic bot that decides whether your pages can be surfaced and cited when Claude answers a query with web sources. Allow Claude-SearchBot if you want visibility in Claude-powered search; it honors robots.txt.
Last updated
- User-agent token
- Claude-SearchBot
- Operator
- Anthropic
- Feeds
- Claude search citations
- robots.txt
- Unverified
How to control Claude-SearchBot in robots.txt
Edit the robots.txt file at the root of your domain (for example https://example.com/robots.txt), add one of the groups below, then save and re-deploy.
Remember: a named User-agent: Claude-SearchBot group overrides your global User-agent: * rules, so repeat any private Disallow paths inside it.
Allow Claude-SearchBot (recommended for AI visibility)
# Welcome Claude-SearchBot, but keep private areas blocked.
# A named user-agent group overrides "User-agent: *", so repeat
# your own private Disallow rules inside this group.
User-agent: Claude-SearchBot
Allow: /
Disallow: /admin/
Disallow: /account/
Disallow: /cart/
Disallow: /checkout/Block Claude-SearchBot
# Block Claude-SearchBot from the entire site.
User-agent: Claude-SearchBot
Disallow: /FAQ
What happens if I block Claude-SearchBot?
Anthropic won't index your content for Claude's search, which can reduce your visibility and the accuracy of Claude-powered search answers about your site.
Is Claude-SearchBot the same as ClaudeBot?
No. ClaudeBot is for training; Claude-SearchBot is for search indexing and citation. They are separate robots.txt tokens.
Is your site visible to AI crawlers?
Run a free AI-visibility audit to see which AI crawlers can reach your content and how often you get cited by ChatGPT, Perplexity, Claude, and Google AI Overviews.
Run a free auditPart of the Cite Hustle AI crawler directory. For the full framework on AI search visibility, read the GEO methodology.