Definition: What is Google-Extended?

Google-Extended is a robots.txt control token introduced by Google in September 2023. Deliberately, it is not a separate user agent but a control signal. According to Google's current documentation, it governs two things: whether crawled content may be used to train future Gemini models, and whether it may be used for grounding, i.e. content from the search index that Gemini Apps and Grounding with Google Search on Vertex AI retrieve at prompt time. Crawling still happens with Google's existing user agents; Google-Extended only governs downstream data usage.

The goal: give publishers a granular choice without forcing the binary "block Googlebot or not". Whoever sets Google-Extended to Disallow stays in search rankings and remains AI-Overview-eligible (see below) - while blocking the use of their content for future Gemini training and for grounding in Gemini Apps and Vertex AI. Google is explicit: Google-Extended does not affect inclusion in Google Search and is not a ranking signal. The token is Google's response to the publisher debate that arose around GPTBot and the training question in 2023.

Core idea

Google-Extended separates Gemini training and grounding from search, not from AI Overviews

The block covers Gemini training and grounding in Gemini Apps and Vertex AI. It does not stop use in AI Overviews and AI Mode, which are part of Google Search. For those features, Google controls display via nosnippet, data-nosnippet, max-snippet or noindex.

Google-Extended vs. Googlebot

The crucial difference is functional, not in the crawl mechanism. Googlebot is the actual crawler that fetches content for the search index. Blocking Googlebot means: the domain disappears from Google Search. Google-Extended is not a crawler; it is a directive that influences Google's internal data-usage pipeline. The content is still crawled (by Googlebot), but not used for AI training.

Operationally, this means a publisher can allow Googlebot (keeping search), block Google-Extended (no training input), and decide on GPTBot separately. The three decisions are independent. For enterprise domains this is the standard matrix in any crawler-policy review.

Syntax and implementation

Google-Extended is configured as a standalone User-agent block in robots.txt. Example for a full block:

User-agent: Google-Extended
Disallow: /

For selective control - blog content trainable, member area blocked:

User-agent: Google-Extended
Allow: /blog/
Allow: /glossary/
Disallow: /customers/
Disallow: /internal/

The order of the blocks in robots.txt is irrelevant - Googlebot, Google-Extended and other user agents are evaluated independently. For full control in the AI era, a combined directive belongs in the standard template:

User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Disallow: /customers/

User-agent: GPTBot
Disallow: /customers/

User-agent: CCBot
Disallow: /

The strategic trade-off question

The decision to block or allow Google-Extended is not technical but strategic. The two positions:

Block: The publisher withholds training input from Google. This protects content investments from being used without direct compensation. The price: your content is available neither for future Gemini training nor for grounding in Gemini Apps. Our working hypothesis: over time this can mean fewer mentions, fewer citations in Gemini and a lower share of model. For paywall publishers and licensing strategies this is acceptable - they monetize through separate contracts.

Allow: The publisher accepts training without direct monetization. The price: content investments are used indirectly. The gain: the brand remains available to Gemini-based answers and co-occurrence with topics builds up; the decision has no effect on inclusion in AI Overviews. For brand-led publishers and B2B trade portals this is the standard choice.

Typical mistakes in Google-Extended strategies

Related terms

Google-Extended belongs to the line of AI crawler and training controls: GPTBot, ClaudeBot, CCBot, PerplexityBot. Technically it is controlled via robots.txt, complemented editorially by llms.txt. For inclusion in AI Overviews, the distinction from Googlebot is decisive.

FAQ on Google-Extended

What is Google-Extended? ▾

Google-Extended is not its own crawler but a robots.txt token introduced by Google in September 2023. It lets publishers control the use of their content for training future Gemini models and for grounding in Gemini Apps and Vertex AI, decoupled from Google Search. Crawling still happens with Google's existing user agents.

How does Google-Extended differ from Googlebot? ▾

Googlebot crawls for Google Search; the index feeds rankings and classical SERPs. Google-Extended is a separate control token for using the same crawled data in AI training. Blocking Google-Extended keeps the domain in normal search but excludes its content from future Gemini training runs.

Does Google-Extended exclude AI Overviews? ▾

No. AI Overviews and AI Mode are part of Google Search, and according to Google, Google-Extended affects neither inclusion in Search nor ranking. To limit how individual content appears in AI features, use nosnippet, data-nosnippet, max-snippet or noindex. Blocking Googlebot would remove the domain from Google Search entirely.

How do you set Google-Extended in robots.txt? ▾

As a User-agent block: User-agent: Google-Extended / Disallow: /. The syntax is identical to other crawlers. Google respects the directive per its own documentation. The block applies to future training runs and to grounding, not retroactively to models already trained.

Should I block Google-Extended? ▾

The trade-off is strategic. Blocking means: your content flows neither into Gemini training nor into grounding in Gemini Apps, which can weaken your presence in Gemini answers over time. Allowing means: training at your expense without direct monetization. For publishers with strong brand positioning, allowing is usually the better choice. For paywall publishers with licensing monetization, blocking is.

Sources