In 2026, the server logs of a typical enterprise website show a new pattern. Alongside Googlebot, Bingbot and the classical SEO bots, new user agents appear with rising frequency: GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, CCBot, Meta-ExternalAgent. An analysis by Vercel and MERJ shows the scale: across Vercel's network, GPTBot made 569 million and Claude 370 million fetches in a month, together about 20% of Googlebot's 4.5 billion fetches in the same period (December 2024). Google-Extended and Applebot-Extended, by contrast, never show up as user agents in your logs: they are robots.txt control tokens only.

The widespread reflex, blanket blocking via robots.txt or Cloudflare rules, is strategically costly. Block GPTBot and you are telling OpenAI: "Please train your next model without our content." That may be legally sensible for publisher properties; for almost every other business model it is a serious long-term visibility risk.

The most relevant AI crawlers at a glance

The overview below shows the most relevant AI crawlers and control tokens in 2026 with operator, purpose and the token for robots.txt and log-file configuration (based on operator documentation, September 2026).

Crawler Operator Training data Live search User-agent string
GPTBotOpenAIyesnoGPTBot
OAI-SearchBotOpenAInoyes (ChatGPT Search)OAI-SearchBot
ChatGPT-UserOpenAInoyes (user-initiated fetches)ChatGPT-User
ClaudeBotAnthropicyesnoClaudeBot
Claude-UserAnthropicnoyes (user-initiated fetches)Claude-User
Claude-SearchBotAnthropicnoyes (search)Claude-SearchBot
PerplexityBotPerplexitynoyes (Perplexity search)PerplexityBot
Perplexity-UserPerplexitynoyes (user-initiated)Perplexity-User
Google-ExtendedGoogleyes (Gemini training & grounding)noGoogle-Extended (robots.txt token only, no user agent of its own)
Applebot-ExtendedAppleyes (Apple foundation models)noApplebot-Extended (robots.txt token only, does not crawl)
CCBotCommon Crawlyes (open crawl corpus, widely used for training)noCCBot
Meta-ExternalAgentMetayes (AI models)noMeta-ExternalAgent
BytespiderByteDanceyesnoBytespider

Note on classification: "Training data: yes" means the crawler collects content for future model iterations; a block only affects your brand's model representation months later. "Live search: yes" marks crawlers that pull content in real time for answers inside search products (ChatGPT Search, Perplexity, Claude): a block removes you immediately from citable sources. This distinction is the strategic basis for any differentiated robots.txt. One caveat: user-initiated fetchers such as ChatGPT-User and Perplexity-User do not necessarily follow robots.txt, according to their operators.

OpenAI: GPTBot and OAI-SearchBot

OpenAI documents three relevant agents in its overview of OpenAI crawlers. GPTBot collects content that may be used to train foundation models. OAI-SearchBot surfaces websites in ChatGPT's search features; according to OpenAI, its content is not used for training. ChatGPT-User fetches pages for user actions; because these are user-initiated, OpenAI says robots.txt rules may not apply. The distinction is fundamental: block GPTBot but allow OAI-SearchBot, and you can still be cited in ChatGPT, but your content will no longer feed model training.

Anthropic: ClaudeBot, Claude-User & Claude-SearchBot

Anthropic currently documents three bots: ClaudeBot for training data, Claude-User for fetches that users trigger in Claude, and Claude-SearchBot for search quality. The older tokens anthropic-ai and Claude-Web no longer appear in the current documentation. According to Anthropic, its bots respect robots.txt and support the non-standard Crawl-delay extension. Anthropic also publishes its IP addresses, but notes that IP blocking alone is not a reliable opt-out.

Google-Extended

A control token for training future Gemini models and for grounding in Gemini Apps and Vertex AI. According to Google, Google-Extended has no user agent of its own: crawling is done with existing Google crawlers, and the token only works in robots.txt. It affects neither inclusion in Google Search nor ranking. Block Google-Extended and your site stays in the search index, and therefore remains a possible source for AI Overviews, which draw on regular indexing.

Apple: Applebot-Extended

Analogous to Google-Extended: an opt-out token that, according to Apple, controls whether content crawled by Applebot may be used to train Apple's foundation models. Applebot-Extended does not crawl itself. Pages that disallow it can still appear in search results in Spotlight, Siri and Safari.

PerplexityBot & CCBot

PerplexityBot collects for Perplexity search and, according to Perplexity, is not used to train foundation models. CCBot is the crawler of the non-profit Common Crawl project, whose open archive serves many LLMs as a training base: a Mozilla Foundation study found that at least 64% of 47 text-generation LLMs examined (2019 to October 2023) used a filtered version of Common Crawl. Blocking CCBot therefore has effects across many models.

Which AI crawlers should you allow, and which should you block?

For most brands the rule is: allow search and retrieval bots, and make a deliberate decision on training crawlers. Block OAI-SearchBot, PerplexityBot or Claude-SearchBot and you lose visibility in the search answers of ChatGPT, Perplexity and Claude. Block GPTBot, ClaudeBot, CCBot or Google-Extended and you only withhold your content from training future models, not from live search.

The operators now separate the two layers clearly. According to OpenAI, sites that opt out of OAI-SearchBot will not appear in ChatGPT search answers, while a GPTBot disallow only affects training. Google states that Google-Extended affects neither inclusion in Google Search nor ranking. Know two limits of robots.txt: user-initiated fetchers such as ChatGPT-User and Perplexity-User do not necessarily follow it, according to their operators. And a disallow only works going forward; Anthropic, for example, says a block excludes a site's future materials from training. Control both layers separately and you stay citable while keeping control over training use.

The specific trade-off depends on the business model. Three main scenarios:

Scenario 1: brands and service providers (default recommendation)

For brands, service providers, B2B vendors and most corporate sites, AI visibility is a marketing asset, not a content loss. Allow every relevant AI crawler, control crawl budget, monitor server load.

Scenario 2: publishers and news outlets

A more complex trade-off space. Blanket blocking protects current content value but costs future relevance in the AI era. A common hybrid course: block GPTBot (training) and allow OAI-SearchBot (live retrieval). You stay citable but prevent training appropriation.

Scenario 3: sensitive or legally exposed content

Pages with personal data, legally protected content or paywalled material: blanket block, plus IP-based rate limits.

Robots.txt: the right configuration

A clean, differentiated robots.txt for the default scenario:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: PerplexityBot
Allow: /

# Sensitive areas
User-agent: *
Disallow: /checkout/
Disallow: /account/
Disallow: /internal/

A frequent mistake: User-agent: * with Disallow: / blocks every crawler, including every AI crawler that respects robots.txt. Differentiation is essential.

Managing crawl budget and server load

AI crawlers create real costs. On an enterprise site with 50,000 pages, our log-file analyses sometimes show several AI bots adding multiple gigabytes of traffic per day. Without control, this leads to:

Practical measures:

Do AI crawlers render JavaScript?

No, most of them do not. In an analysis by Vercel and MERJ, none of the major AI crawlers from OpenAI, Anthropic, Meta, ByteDance and Perplexity rendered JavaScript. Some download JS files but do not execute them. The exceptions are Gemini, which uses Googlebot's infrastructure, and Applebot with browser-based rendering. GPTBot, ClaudeBot and PerplexityBot therefore see only the initially served HTML. Dynamic content loaded client-side via React/Vue/Angular is invisible to these crawlers.

The concrete consequences:

~20%

GPTBot + Claude relative to Googlebot volume (Vercel network, 2024)

0

major AI crawlers rendering JavaScript per Vercel/MERJ (except Gemini and Applebot)

7-14 days

Crawl interval for top pages we often see in audits (own data)

Operator Insight

The status code that hurts

An often-overlooked factor: 429 Too Many Requests and 503 Service Unavailable. For Google it is documented that 429 and 5xx responses slow down crawling and that URLs returning persistent server errors drop out of the index. AI-crawler operators do not document their behaviour here, but in our log files we see markedly fewer visits after error streaks as well. An under-dimensioned server can erode your AI visibility without classical SEO reports catching it.

Structured data: the LLM accelerator

Where classical SEO teams treat schema markup as a CTR booster for rich snippets, structured markup has a more fundamental function in the AI era: it reduces ambiguity for models and raises the probability of correct information extraction.

Especially effective:

Monitoring: what you should measure

A modern technical-SEO monitor actively includes AI crawlers:

The complete robots.txt for differentiated AI-crawler access

A typical production setup for a B2B brand with high reputation interest that also protects monetized archives:

# SUMAX Enterprise Reference Configuration
# Last updated: 2026-03-01

User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Allow: /
Disallow: /members/
Disallow: /internal/

User-agent: GPTBot
Allow: /
Disallow: /members/
Disallow: /pricing-calculator/
Disallow: /internal/

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /
Disallow: /members/

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Allow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: *
Allow: /
Disallow: /cgi-bin/
Disallow: /search?
Disallow: /*?utm_
Disallow: /print/

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-news.xml

The distinctions matter: GPTBot (training) must not see monetizable assets; OAI-SearchBot (live retrieval for ChatGPT Search) sees everything, because that is where citation value is created. Google-Extended controls use for Gemini training and grounding, not inclusion in AI Overviews, which draw on the regular Google index. If you do not want to appear there, Google says to use nosnippet, data-nosnippet, max-snippet or noindex, not Google-Extended. Confusing these two layers was one of the most common strategic mistakes of 2024.

Log-file analysis: the operational gold standard

Crawler behaviour cannot be measured with SEO tools, only with server logs. A minimal setup for AI-crawler analysis:

# Extract AI-crawler hits from Apache/Nginx log (awk/cut)
# Fields: IP, UserAgent, Status, Path, Timestamp

# Note: Google-Extended and Applebot-Extended never appear in logs (robots.txt tokens only)
grep -E "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot" access.log \
  | awk '{print $1, $7, $9, $NF}' \
  > ai_crawler_hits.tsv

# Aggregation: hits per bot per day per path pattern
# Target metrics:
#   - hit rate per path cluster (/blog/*, /product/*, /case-study/*)
#   - 2xx rate per bot (target > 97%)
#   - median response time per bot (target < 600 ms)
#   - re-crawl interval (median delta between two hits of the same path)

A healthy crawl pattern for enterprise domains (our own data from our benchmarks and audits, evidence level C):

~1,240

GPTBot requests/day, median for enterprise domains with more than 10,000 URLs

3-7 days

Re-crawl interval for top content on healthy domains

> 97%

Target 2xx rate per AI crawler

JavaScript rendering in practice: consequences and fixes

As things stand, AI crawlers render little or no JavaScript. They read only the initial HTML document, so rely exclusively on server-delivered HTML. Anything loaded client-side simply does not exist for them.

Practical consequences:

Solutions, ranked by effort:

  1. Activate SSR. Next.js, Nuxt, Remix ship static HTML out of the box. Minimum effort, maximum effect.
  2. Dynamic rendering (server-side renders for bots, client-side for users). Google also calls it a workaround and recommends server-side rendering, static rendering or hydration instead.
  3. Prerendering. Static HTML snapshots on the CDN served on bot detection. Tools such as Prerender.io; the once-popular Rendertron has been archived since 2022.
  4. Content migration to MDX/Markdown sources with static build. The cleanest solution for content platforms.

Rate limiting, CDN policy and the 429 dead-zone effect

Aggressive WAF/CDN rules (Cloudflare, Akamai, Fastly) often block AI crawlers unnoticed. Typical scenario: the WAF sees an unusual user-agent pattern, classifies it as bot traffic, throttles to 10 req/min. GPTBot hits the limit, receives 429 Too Many Requests and, in our observation, often visits less for weeks. The domain disappears from LLM outputs even though robots.txt is clean.

Controls:

Sitemap strategy: separate signals for separate purposes

A single sitemap.xml is often no longer sufficient for large sites. We recommend a three-sitemap structure (our recommendation, evidence level D):

  1. sitemap-core.xml: canonical, durable URLs with an accurate lastmod; Google ignores changefreq and priority. For training crawlers.
  2. sitemap-news.xml: news format with publication node. For OAI-SearchBot, PerplexityBot. Dynamic; per Google's rules, only articles from the last two days.
  3. sitemap-knowledge.xml: definitional/evergreen content (pillar pages, glossary, studies). Especially important for LLM training.

The split helps crawlers prioritize content by lifecycle and purpose. In our log-file analyses, training crawlers such as GPTBot fetch evergreen content disproportionately, while live-retrieval bots such as OAI-SearchBot lean more towards recent content; OpenAI does not document this behaviour. A monolithic sitemap forces identical prioritization on both scenarios, which is suboptimal.

Monitoring dashboard: what gets reviewed weekly

Technical AI-crawler governance needs its own dashboard. Six core metrics:

Operator Insight

The invisible 15% domain

Our audits regularly reveal enterprise domains where 15-30% of all URLs are effectively unreachable for AI crawlers, not because of robots.txt, but because of WAF throttling, outdated SSL configuration or false JS-rendering assumptions. This gap is often unknown internally because classical SEO tools do not surface it. Only the combination of log-file analysis, prompt audit and infrastructure check exposes it.

Conclusion

Technical SEO is not a settled topic in the AI era; it is a strategically upgraded field. The infrastructure decisions you make today determine whether your brand is stored as a reliable source in the next model generations or remains a fragmented, contradictory entity in the noise.

Blanket blocking may feel defensively correct. For most business models it is a strategic self-limitation with a long downstream effect.

Sources