Anthropic · User-initiated fetcher
Claude-User
Claude-User retrieves a page when a person asks Claude a question that requires reading that specific URL. Like OpenAI's ChatGPT-User, it is a user-initiated fetcher rather than a bulk crawler, and Anthropic confirms it honors robots.txt. Blocking Claude-User stops Claude from fetching your pages in response to user queries.
Last updated
- User-agent token
- Claude-User
- Operator
- Anthropic
- Feeds
- Claude live page fetches
- robots.txt
- Unverified
How to control Claude-User in robots.txt
Edit the robots.txt file at the root of your domain (for example https://example.com/robots.txt), add one of the groups below, then save and re-deploy.
Remember: a named User-agent: Claude-User group overrides your global User-agent: * rules, so repeat any private Disallow paths inside it.
Allow Claude-User (recommended for AI visibility)
# Welcome Claude-User, but keep private areas blocked.
# A named user-agent group overrides "User-agent: *", so repeat
# your own private Disallow rules inside this group.
User-agent: Claude-User
Allow: /
Disallow: /admin/
Disallow: /account/
Disallow: /cart/
Disallow: /checkout/Block Claude-User
# Block Claude-User from the entire site.
User-agent: Claude-User
Disallow: /FAQ
Does Claude-User crawl my whole site?
No. It fetches individual pages on demand when a Claude user's question requires them. It is not used for training or bulk indexing.
Does Claude-User follow robots.txt?
Yes — Anthropic's February 2026 documentation states Claude-User honors robots.txt, alongside ClaudeBot and Claude-SearchBot.
Is your site visible to AI crawlers?
Run a free AI-visibility audit to see which AI crawlers can reach your content and how often you get cited by ChatGPT, Perplexity, Claude, and Google AI Overviews.
Run a free auditPart of the Cite Hustle AI crawler directory. For the full framework on AI search visibility, read the GEO methodology.