Definition: What is llms.txt?
llms.txt is a format proposed by Jeremy Howard (Answer.AI) on September 3, 2024 for a curated, Markdown-formatted file at the website root (/llms.txt). Version 2 of the proposal was published on August 10, 2026. It gives large language models and their tool stacks a machine-readable overview of a site's most important content: core pages, documentation, API references, whitepapers, policy pages. The format is deliberately lean - Markdown, no new schemata - and is therefore easy to produce for publishers and to consume for LLMs.
An important framing: llms.txt is a proposal, not a formal web standard. Neither the IETF nor the W3C has ratified it, and no major provider has confirmed that its search crawlers evaluate the file. In June 2026 Google even documented explicitly that Google Search does not use llms.txt: the file neither helps nor hurts visibility or rankings, including in AI Overviews and AI Mode (Google's guide). The format is most common in developer documentation: according to llmstxt.org, OpenAI, Anthropic and Google (Gemini) publish llms.txt files for their own developer docs, and Chrome's Lighthouse checks for the file as part of its agentic browsing audits. Its value lies with agents and coding tools that read websites deliberately, not in search visibility.
llms.txt is a signpost for agents, not a ranking signal
The file regulates nothing. It curates. According to Google it has no effect on Google Search; its value lies with agents and developer tools that look for it, and in the editorial process: which content really matters to an external machine?
Distinction from robots.txt
The two formats are often confused. robots.txt is an access-control mechanism: which crawlers may fetch which paths. It emerged in 1994 as a de-facto standard, was formally standardized in 2022 as RFC 9309 and is respected by every serious crawler. llms.txt, by contrast, is a curation recommendation: which content is particularly relevant for LLM consumption. It regulates nothing. Both files sit at the root but have fundamentally different functions.
Operationally they complement each other: GPTBot, Google-Extended and ClaudeBot are regulated via robots.txt; llms.txt curates content for cases where the crawlers are allowed - or where users open the site directly in a chatbot.
Structure and syntax
The format is Markdown. According to the specification only an H1 with the site name is required; the recommended structure is:
# Site name > Short description of the site in 1-3 sentences. Optional extended section with additional context, audience, license or usage notes. ## Docs - [Getting Started](https://example.com/docs/start): First-time setup - [API Reference](https://example.com/docs/api): All endpoints ## Examples - [Quickstart](https://example.com/examples/quickstart.md): Minimal example - [Advanced](https://example.com/examples/advanced.md): Advanced patterns ## Optional - [Changelog](https://example.com/changelog) - [Archive](https://example.com/archive)
The ## Optional section collects secondary links by convention. In version 1, tools could skip it when context was tight; version 2 keeps it as a convention but without that mechanical meaning.
New in version 2: pages with information agents might need should offer a clean Markdown version at the same URL (page.html.md or page.md). The Markdown version and the responsible llms.txt become discoverable via the link relations rel="alternate" type="text/markdown" and rel="describedby", as HTML elements or an HTTP Link header. An llms.txt in a subdirectory such as /docs/llms.txt covers all pages below it.
llms-full.txt: convention, not specification
You will often come across a second file: /llms-full.txt. It is not part of the specification but a widespread convention, mainly from documentation platforms. It contains the plain text of all pages referenced from llms.txt in a single file so that a tool can load the site context with one request. Version 2 of the proposal instead expects agents to read the llms.txt and follow the linked Markdown pages as needed.
For enterprise sites with thousands of pages, a complete llms-full.txt is not realistic; we recommend prioritizing the 30 to 100 most important pages. Documentation sites remain the most common use case.
Does Google use llms.txt?
No. On June 15, 2026, Google clarified in its documentation changelog that llms.txt files are not needed for Google Search and neither improve nor harm visibility or rankings. That explicitly includes AI Overviews and AI Mode. According to Google, it is still fine to maintain the file if other services use it.
In practice this means llms.txt is not an SEO or GEO lever for Google. It remains useful for developer documentation, agents and coding tools that look for the file. For visibility in Google Search, crawlability, indexing and helpful content are what count.
Practice: llms.txt for enterprise sites
Operational rollout in six steps:
- Curation: which 30-50 pages represent the core expertise and core services? Decision jointly with editorial and product.
- Categorization: sections like
## Services,## Methodology,## Cases,## Glossary. - Descriptions: every link with a 5-15-word summary. No marketing platitudes, concrete facts.
- Optional section: blog archives, press archives, deep feature pages.
- Maintenance: quarterly update. Remove dead links. Add new core content islands.
- Technical: file as UTF-8, content type
text/plain; charset=utf-8, HTTP 200 on/llms.txt. No redirect, no CMS wrapper.
Typical mistakes in llms.txt implementations
- Auto-generated dump. Every URL from the sitemap copied 1:1. Defeats the core purpose of curation. llms.txt loses its signal value.
- Marketing bullet points instead of facts. "We are the market leader". Without concrete substantive value. LLMs do not use this as context.
- No maintenance. A file from January 2024, dead links to restructured pages. Trust signal negative.
- SEO expectations. Planning llms.txt as a ranking lever for Google. Google Search ignores the file.
- Confusion with robots.txt. Access blocks in llms.txt have no effect. Whoever wants to block GPTBot does it in robots.txt.
- Oversized llms-full.txt. Very large files exceed the context windows of many models and are only partly read. Prioritize true core content.
Related terms
llms.txt belongs to the operational AI crawler setup - together with robots.txt, GPTBot, Google-Extended and other AI crawler user agents. Strategically it is part of GEO and LLM SEO infrastructure. For documentation sites it complements XML sitemaps as a semantic curation signal.
FAQ on llms.txt
What is llms.txt? ▾
llms.txt is a format proposed by Jeremy Howard (Answer.AI) in September 2024, in version 2 since August 2026, for a curated, Markdown-formatted overview of a website's most important content. The goal: give AI agents and language models a navigation aid so they can find relevant pages directly.
Is llms.txt an official standard? ▾
No. llms.txt is a proposal without formal ratification by the IETF or W3C. No major provider has confirmed that its search crawlers evaluate the file; in June 2026 Google documented that Google Search ignores it. The format is most common in developer documentation.
How does llms.txt differ from robots.txt? ▾
robots.txt regulates crawler access - what may be crawled. llms.txt curates content - what is relevant. robots.txt is access control, llms.txt is a recommendation. Both files live at the root, but they have different functions and are consumed by different systems.
How is llms.txt structured? ▾
Markdown format with an H1 title (site name), a blockquote summary, optional detail paragraphs, and H2 sections containing grouped links with descriptions. Version 2 adds Markdown versions of pages and link relations for discoverability. The widespread llms-full.txt is a convention, not part of the specification.
Do I have to create llms.txt? ▾
Not for Google Search: Google ignores the file. It makes sense for knowledge bases, developer documentation and trade portals whose content agents and coding tools should use. A side benefit: llms.txt forces you to curate your most important pages.