Definition and user agents
Perplexity documents two separate access types:
- PerplexityBot: the index crawler collecting content for search; user-agent string with
PerplexityBotand a verification URL, published IP ranges. - Perplexity-User: live fetches when a user asks a specific question or opens a URL — user-initiated and, per documentation, executed even when the index crawler is blocked.
Perplexity is designed as a RAG system: every answer synthesises from retrieval results and displays its sources visibly. Documented citation rates are higher there than in any other generative system — 2 to 8 percent for domain authorities on subject-matter queries (working values from ongoing measurement).
The crawling controversy
Honest framing includes the backstory: in 2024, several publishers and infrastructure providers documented fetches from Perplexity services outside the declared crawler that bypassed robots.txt blocks — followed by public disputes and legal conflicts with publishers. Perplexity has since expanded its user-agent documentation, launched publisher programmes and clarified the separation of index crawling and user-initiated fetches. If you really want to control access, do not rely on robots.txt alone — check logfiles and the published IP ranges.
Control and practice
# allow the Perplexity index User-agent: PerplexityBot Allow: /
- Visibility trade-off: Perplexity cites prominently and links clickably — for most professional and B2B brands the visibility benefit prevails.
- Logfile check: confirm real PerplexityBot hits via IP verification; hunt 403/429 patterns in WAF rules.
- Citation monitoring: measure your own citation rate in Perplexity against competitors regularly — the system reacts comparatively quickly to fresh, structured sources.
Related terms
PerplexityBot stands beside GPTBot, OAI-SearchBot and ClaudeBot in the crawler policy of the retrieval layer. The optimisation analysis for the system: Perplexity SEO.
The most citation-friendly system has its own rules
Perplexity cites more sources per answer than any other system — and has a documented crawling history. If you want visibility, open the crawler; if you want control, check logs, not just robots.txt.
FAQ on PerplexityBot
What is the difference between PerplexityBot and Perplexity-User? ▾
PerplexityBot is the index crawler for search. Perplexity-User fetches pages live when a user asks a specific question or opens a URL — per documentation even when PerplexityBot is blocked via robots.txt, because the fetch counts as user-initiated.
Why is Perplexity especially relevant for GEO? ▾
Because it is the most citation-friendly generative system: every answer displays sources visibly and clickably, and measured citation rates exceed those of ChatGPT and AI Overviews. For professional and B2B brands, Perplexity is often the fastest proof channel for GEO effect.
Does Perplexity respect robots.txt? ▾
For the declared PerplexityBot: yes, per documentation. In 2024, however, publishers and infrastructure providers documented fetches outside the declared crawler; Perplexity has since expanded documentation and publisher programmes. Reliable control comes from logfile analysis plus IP verification, not robots.txt alone.
How do you increase citations in Perplexity? ▾
With the same mechanics as for all RAG systems, only visible faster: fresh, dated content, passage-precise structured answers, a clear author entity and unrestricted crawler access. Measure progress via your citation rate against competitors.