Definition: What is GPTBot?

GPTBot is OpenAI's official web crawler. It was introduced in August 2023 and collects publicly accessible content for future training runs of the GPT model family. The user agent is simply GPTBot; the full user-agent string is listed in OpenAI's crawler documentation, and the official IP ranges are published as a JSON file at openai.com/gptbot.json. Thanks to published IP ranges the crawler can be reliably distinguished from spoofing attempts.

An important distinction: GPTBot is not OpenAI's only user agent. For the ChatGPT Search function, OAI-SearchBot operates; for live fetches during a user session, ChatGPT-User. The three user agents have different purposes and are addressed separately in robots.txt; for the user-initiated ChatGPT-User, OpenAI notes that robots.txt rules may not apply. Blocking "GPTBot" does not automatically block ChatGPT Search - a common misunderstanding in enterprise policies.

Core idea

The most common blocking problem is not robots.txt but the WAF

Many domains allow GPTBot in robots.txt - but block it via Cloudflare, Akamai or AWS WAF with 429 rate limits. The result: content is missing from future training data. If the same rule also hits OAI-SearchBot, it is missing from ChatGPT Search as well.

Control via robots.txt

The primary control happens via the robots.txt file at the website root. Basic syntax for a full block:

User-agent: GPTBot
Disallow: /

For selective control - opening only the blog, blocking the members area:

User-agent: GPTBot
Allow: /blog/
Disallow: /

For allowing with fine-grained control across multiple OpenAI user agents:

User-agent: GPTBot
Disallow: /internal/
Disallow: /customers/

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

According to OpenAI's documentation, GPTBot follows robots.txt: a Disallow signals that the content should not be used to train generative AI foundation models. The rule applies from the next fetch onward, not retroactively to models already trained.

The 429 problem: WAF blocks despite an open robots.txt

In practice GPTBot fails on many domains not at robots.txt but at the Web Application Firewall. Cloudflare, Akamai, AWS WAF, Imperva and Fastly carry default rules that aggressively rate-limit bot traffic from unknown user agents or respond with challenge prompts. GPTBot cannot solve CAPTCHAs and consequently receives HTTP 429 Too Many Requests or 403 Forbidden - even though the robots.txt directive explicitly allows it.

We see the pattern often in audits, and it has intensified: since July 1, 2025, Cloudflare asks every newly onboarded domain whether to allow AI crawlers and starts from controlled access rather than an open default. Anyone who does not decide consciously during setup may block AI crawlers unintentionally. For publishers who value LLM visibility this is an operational problem: content is missing from future training data even though robots.txt allows it.

Practice: WAF policy for GPTBot

  1. Verification. Pull the official IP range from OpenAI's JSON list (openai.com/gptbot.json) and mark it as trusted.
  2. Raise rate limits. Allow at least 100 requests/minute for verified GPTBot IPs.
  3. Disable challenges. No JavaScript or CAPTCHA challenges for GPTBot IPs. According to an analysis by Vercel and MERJ (December 2024), the crawler does not execute JavaScript.
  4. Logging. Use a separate log category for GPTBot traffic. Monthly check of 4xx/5xx response rates. Target: < 2 percent 429/403.
  5. Documentation. Record the WAF exception in the security policy so it does not get wiped during routine updates.

In parallel you decide which areas to keep blocked via robots.txt: members areas, internal tools, customer dashboards. Public content - blog, glossary, product pages, press area - typically belongs in the allowed zone.

Typical mistakes in GPTBot strategies

Related terms

GPTBot is part of the AI crawler landscape alongside Google-Extended, ClaudeBot, PerplexityBot and CCBot. Control happens via robots.txt, supplemented by llms.txt and WAF policy. Strategically, GPTBot belongs in any GEO and LLM SEO infrastructure.

FAQ on GPTBot

What is GPTBot? ▾

GPTBot is OpenAI's web crawler, introduced in August 2023. It collects publicly accessible web content as training material for future GPT model generations. The user agent is simply 'GPTBot'. In parallel, OAI-SearchBot operates for the ChatGPT Search functionality and ChatGPT-User for live fetches during a user session.

How do you block GPTBot? ▾

Via an entry in robots.txt: User-agent: GPTBot / Disallow: /. OpenAI documents this officially and respects the directive. The block applies to future crawling; already trained models retain the content. For full control, also address OAI-SearchBot and ChatGPT-User; for ChatGPT-User, OpenAI notes that robots.txt rules may not apply because the fetches are user-initiated.

What is the 429 problem? ▾

Many sites unintentionally block GPTBot via WAF rules or rate limits that produce 429 Too Many Requests. The crawler is then classified as suspicious traffic and discarded, even though robots.txt allows access. In our audits this is among the most common reasons content does not reach OpenAI despite an open robots.txt; if the rule also hits OAI-SearchBot, the pages are missing from ChatGPT Search as well.

Does GPTBot differ from OAI-SearchBot? ▾

Yes. GPTBot collects training data for model updates. OAI-SearchBot indexes for the ChatGPT Search functionality - comparable to a classical search crawler. ChatGPT-User, in turn, fetches content live on a user request. All three user agents are addressed separately in robots.txt; for ChatGPT-User, OpenAI notes the rules may not apply.

Does allowing GPTBot harm your own domain? ▾

As a rule, no. GPTBot follows robots.txt, uses published IP ranges (openai.com/gptbot.json) and produces manageable load. Blocking signals to OpenAI that the content should not be used for future training. For publishers who value LLM visibility, allowing it is the standard recommendation.

Sources