Definition and context
OpenAI operates three documented user agents with clearly separated jobs:
- OAI-SearchBot: crawls for the ChatGPT Search index — the basis for citations and links in answers with web search.
- GPTBot: collects training data for future models.
- ChatGPT-User: fetches pages live when a user triggers a specific URL or research task in ChatGPT.
The user-agent string contains OAI-SearchBot plus a verification URL; OpenAI publishes the corresponding IP ranges for authenticity checks. According to OpenAI's documentation, all three crawlers respect robots.txt.
The visibility consequence
Blocking OAI-SearchBot removes you from the source pool of ChatGPT Search — with, per OpenAI figures (as of 2025), over 800 million weekly users and a growing share of search-like prompts, a consequential decision. What matters is separating the decisions: a brand can block training (GPTBot) and allow search (OAI-SearchBot) — or vice versa. Blanket blocking of “all AI crawlers” conflates three different business questions.
# allow search, block training User-agent: OAI-SearchBot Allow: / User-agent: GPTBot Disallow: /
Practice checks
- Check robots.txt: explicit rules per crawler instead of wildcard assumptions.
- WAF and bot protection: many blocks happen not in robots.txt but in firewalls and CDN rules — log analysis and status-code tests (200 instead of 403/429) belong in the audit.
- Verification: distinguish real OAI-SearchBot hits from fake crawlers via the published IP ranges.
- Monitoring: watch crawl frequency in the logs — it shows whether and how deeply the index covers the domain.
Related terms
OpenAI's crawler trio stands beside PerplexityBot, ClaudeBot and Google-Extended. Crawler policy is part of the retrieval layer of GEO; curated content hints are added by llms.txt. Full analysis: Technical SEO for AI crawlers.
Three OpenAI crawlers, three separate decisions
OAI-SearchBot (search), GPTBot (training) and ChatGPT-User (live fetch) answer different business questions. Blanket blocking decides your ChatGPT visibility without you noticing.
FAQ on OAI-SearchBot
What is the difference between OAI-SearchBot and GPTBot? ▾
OAI-SearchBot crawls for the ChatGPT Search index — it determines whether a domain can be cited and linked in answers. GPTBot collects training data for future models. Both can be controlled separately in robots.txt.
Should you block OAI-SearchBot? ▾
For most business models, no: blocking removes the domain from ChatGPT Search's source pool and costs citation visibility. It can make sense for paid-content models whose value an answer would pre-empt — that is a business decision, not an SEO reflex.
Does OAI-SearchBot respect robots.txt? ▾
According to OpenAI's documentation, yes — as do GPTBot and ChatGPT-User. OpenAI additionally publishes the crawlers' IP ranges so real hits can be verified and fake crawlers separated.
How do I check whether ChatGPT can reach my pages? ▾
In three steps: check robots.txt for explicit rules, analyse server logs for OAI-SearchBot hits with status 200 (403/429 indicate WAF blocks), and spot-test answers in ChatGPT Search for your own citations.