Free tool
robots.txt Generator for AI Crawlers
This generator builds a robots.txt that controls which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more) can access your site. It carries the private paths you list into every crawler group, because a named user-agent group overrides your global rules: if you allow a bot without repeating your Disallow rules, you would expose those private paths to it. Choose which bots to allow or block, list any private paths, then copy the result to /robots.txt.
These are repeated inside every crawler group, because a named user-agent group overrides User-agent: *.
robots.txt
How to use it
-
1
List the private paths you never want crawled (one per line, for example /admin/).
-
2
Choose Allow or Block for each AI crawler.
-
3
Copy the generated robots.txt and upload it to your domain root as /robots.txt.
FAQ
Why does the generator repeat my Disallow rules in every crawler group?
Because in the robots.txt standard a named "User-agent: GPTBot" group overrides the global "User-agent: *" group entirely. If a named group omits your private Disallow rules, that bot is no longer bound by them, so the generator repeats them to prevent an accidental leak.
Does blocking a crawler remove me from that AI product?
It depends on the bot's role. Blocking a training crawler keeps you out of model training; blocking a search indexer removes you from that engine's AI search citations. See the AI crawlers directory for what each bot feeds.
Is robots.txt enough to block every AI crawler?
Well-behaved crawlers honor robots.txt, but a few have disputed or unverified compliance. For those, enforce the block at your server or WAF as well. robots.txt is the baseline, not a guarantee.
Check the technical access surface.
These tools inspect individual inputs. Run the free technical access audit for a bounded crawler-controls and raw-HTML score plus prioritized observations. Neither result claims to measure indexing, rankings, or citations.
Run a free audit