Perplexity · Search indexer

PerplexityBot

PerplexityBot is Perplexity's declared crawler that indexes pages so they can be surfaced and cited in Perplexity's answers. Perplexity's help documentation states PerplexityBot will not index sites that disallow it in robots.txt. That claim has been contested: Cloudflare reported in 2024 and again in 2026 that Perplexity sometimes fetched blocked content using stealth, stock-browser user agents, which Perplexity disputed.

Last updated July 2026

User-agent token: PerplexityBot
Operator: Perplexity
Feeds: Perplexity search index
robots.txt: Unverified

Perplexity states PerplexityBot honors robots.txt; this was disputed by Cloudflare in 2024–2026.

How to control PerplexityBot in robots.txt

Edit the robots.txt file at the root of your domain (for example https://example.com/robots.txt), add one of the groups below, then save and re-deploy. Remember: a named User-agent: PerplexityBot group overrides your global User-agent: * rules, so repeat any private Disallow paths inside it.

Allow PerplexityBot (recommended for AI visibility)

# Welcome PerplexityBot, but keep private areas blocked.
# A named user-agent group overrides "User-agent: *", so repeat
# your own private Disallow rules inside this group.
User-agent: PerplexityBot
Allow: /
Disallow: /admin/
Disallow: /account/
Disallow: /cart/
Disallow: /checkout/

# Welcome PerplexityBot, but keep private areas blocked.
# A named user-agent group overrides "User-agent: *", so repeat
# your own private Disallow rules inside this group.
User-agent: PerplexityBot
Allow: /
Disallow: /admin/
Disallow: /account/
Disallow: /cart/
Disallow: /checkout/

Block PerplexityBot

# Block PerplexityBot from the entire site.
User-agent: PerplexityBot
Disallow: /

# Block PerplexityBot from the entire site.
User-agent: PerplexityBot
Disallow: /

FAQ

Does PerplexityBot obey robots.txt?

Perplexity says yes — its documentation states PerplexityBot won't index content disallowed in robots.txt. Cloudflare has publicly disputed this, alleging undeclared crawlers, so treat a robots.txt block as necessary but not always sufficient and verify with server logs.

What is the difference between PerplexityBot and Perplexity-User?

PerplexityBot builds Perplexity's search index. Perplexity-User is the user-initiated fetcher that retrieves a page to answer a specific live query.

Is your site visible to AI crawlers?

Run a free AI-visibility audit to see which AI crawlers can reach your content and how often you get cited by ChatGPT, Perplexity, Claude, and Google AI Overviews.

Run a free audit

Part of the Cite Hustle AI crawler directory. For the full framework on AI search visibility, read the GEO methodology.