Free tool
robots.txt Generator for AI Crawlers
This generator builds a robots.txt that controls which AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more — can access your site. It carries the private paths you list into every crawler group, because a named user-agent group overrides your global rules: if you allow a bot without repeating your Disallow rules, you would expose those private paths to it. Choose which bots to allow or block, list any private paths, then copy the result to /robots.txt.
These are repeated inside every crawler group, because a named user-agent group overrides User-agent: *.
robots.txt
How to use it
-
1
List the private paths you never want crawled (one per line, for example /admin/).
-
2
Choose Allow or Block for each AI crawler.
-
3
Copy the generated robots.txt and upload it to your domain root as /robots.txt.
FAQ
Why does the generator repeat my Disallow rules in every crawler group?
Because in the robots.txt standard a named "User-agent: GPTBot" group overrides the global "User-agent: *" group entirely. If a named group omits your private Disallow rules, that bot is no longer bound by them — so the generator repeats them to prevent an accidental leak.
Does blocking a crawler remove me from that AI product?
It depends on the bot's role. Blocking a training crawler keeps you out of model training; blocking a search indexer removes you from that engine's AI search citations. See the AI crawlers directory for what each bot feeds.
Is robots.txt enough to block every AI crawler?
Well-behaved crawlers honor robots.txt, but a few have disputed or unverified compliance. For those, enforce the block at your server or WAF as well — robots.txt is the baseline, not a guarantee.
See how AI search sees your site.
These tools fix the inputs. Run a free AI-visibility audit to measure the output — how often ChatGPT, Perplexity, Claude, and Google AI Overviews actually cite you.
Run a free audit