Google · Search indexer
Googlebot
Googlebot is Google's core search crawler, and the same index it builds also powers AI Overviews and AI Mode. There is no separate crawler for AI Overviews — if Googlebot can index a page, that page is eligible to be summarized in an AI Overview. You cannot opt out of AI Overviews without affecting normal Search; instead you limit how your content is used with snippet controls like nosnippet, data-nosnippet, and max-snippet.
Last updated
- User-agent token
- Googlebot
- Operator
- Feeds
- Google Search + AI Overviews
- robots.txt
- Unverified
How to control Googlebot in robots.txt
Edit the robots.txt file at the root of your domain (for example https://example.com/robots.txt), add one of the groups below, then save and re-deploy.
Remember: a named User-agent: Googlebot group overrides your global User-agent: * rules, so repeat any private Disallow paths inside it.
Allow Googlebot (recommended for AI visibility)
# Welcome Googlebot, but keep private areas blocked.
# A named user-agent group overrides "User-agent: *", so repeat
# your own private Disallow rules inside this group.
User-agent: Googlebot
Allow: /
Disallow: /admin/
Disallow: /account/
Disallow: /cart/
Disallow: /checkout/Block Googlebot
# Block Googlebot from the entire site.
User-agent: Googlebot
Disallow: /FAQ
How do I stop my content appearing in Google AI Overviews?
There is no AI-Overviews-only opt-out. Blocking Googlebot removes you from Search entirely. To limit usage, apply the nosnippet or max-snippet robots meta directives, which restrict the text Google can show — including in AI Overviews.
Does Google-Extended control AI Overviews?
No. Google-Extended governs Gemini/Vertex generative training and grounding, not AI Overviews. AI Overviews are built from the standard Search index that Googlebot crawls.
Is your site visible to AI crawlers?
Run a free AI-visibility audit to see which AI crawlers can reach your content and how often you get cited by ChatGPT, Perplexity, Claude, and Google AI Overviews.
Run a free auditPart of the Cite Hustle AI crawler directory. For the full framework on AI search visibility, read the GEO methodology.