llms.txt

Definition: What is llms.txt?

llms.txt is a standard proposed in 2024 by Jeremy Howard (Answer.AI) for a curated, Markdown-formatted file at the website root (/llms.txt). It gives large language models and their tool stacks a machine-readable overview of a site's most important content: core pages, documentation, API references, whitepapers, policy pages. The format is deliberately lean - Markdown, no new schemata - and is therefore easy to produce for publishers and to consume for LLMs.

An important framing: llms.txt is a proposed standard, not a formal web standard. Neither the IETF nor the W3C have ratified it. Neither OpenAI, Anthropic, Google nor Perplexity have officially confirmed that their crawlers consume the file normatively. Adoption is community-driven: Answer.AI, the Anthropic documentation, Vercel, Stripe and numerous dev-tool providers have implemented llms.txt. The strategic value lies in editorial curation and signal - not in guaranteed LLM impact.

Distinction from robots.txt

The two formats are often confused. robots.txt is an access-control mechanism: which crawlers may fetch which paths. It is a de-facto standard established in 1994, respected by every serious crawler. llms.txt, by contrast, is a curation recommendation: which content is particularly relevant for LLM consumption. It regulates nothing. Both files sit at the root but have fundamentally different functions.

Operationally they complement each other: GPTBot, Google-Extended and ClaudeBot are regulated via robots.txt; llms.txt curates content for cases where the crawlers are allowed - or where users open the site directly in a chatbot.

Structure and syntax

The format is Markdown. The specification recommends the following structure:

# Site name

> Short description of the site in 1-3 sentences.

Optional extended section with additional context,
audience, license or usage notes.

## Docs

- [Getting Started](https://example.com/docs/start): First-time setup
- [API Reference](https://example.com/docs/api): All endpoints

## Examples

- [Quickstart](https://example.com/examples/quickstart.md): Minimal example
- [Advanced](https://example.com/examples/advanced.md): Advanced patterns

## Optional

- [Changelog](https://example.com/changelog)
- [Archive](https://example.com/archive)

The ## Optional section is special: links there are hints that LLM consumers may skip when the token budget is tight. Every other section counts as important.

llms-full.txt - the full-text variant

Alongside the curated link list, the specification recommends a second file: /llms-full.txt. It contains the complete plain text of all pages referenced from llms.txt, concatenated in a tidy way. Goal: an LLM can load the relevant site context into its context window with a single request, without fetching each page separately. That reduces latency and tool-call overhead dramatically in dev-agent usage.

For enterprise sites with thousands of pages, llms-full.txt is not realistic. Prioritization on the 30-100 most important pages is the standard. For documentation sites (Stripe, Vercel, the Anthropic docs) the full version is the primary use case.

Practice: llms.txt for enterprise sites

Operational rollout in six steps:

1. Curation: which 30-50 pages represent the core expertise and core services? Decision jointly with editorial and product.
2. Categorization: sections like ## Services, ## Methodology, ## Cases, ## Glossary.
3. Descriptions: every link with a 5-15-word summary. No marketing platitudes, concrete facts.
4. Optional section: blog archives, press archives, deep feature pages.
5. Maintenance: quarterly update. Remove dead links. Add new core content islands.
6. Technical: file as UTF-8, content type text/plain; charset=utf-8, HTTP 200 on /llms.txt. No redirect, no CMS wrapper.

Typical mistakes in llms.txt implementations

• Auto-generated dump. Every URL from the sitemap copied 1:1. Defeats the core purpose of curation. llms.txt loses its signal value.
• Marketing bullet points instead of facts. "We are the market leader". Without concrete substantive value. LLMs do not use this as context.
• No maintenance. A file from January 2024, dead links to restructured pages. Trust signal negative.
• Confusion with robots.txt. Access blocks in llms.txt have no effect. Whoever wants to block GPTBot does it in robots.txt.
• Oversized llms-full.txt. Files over 5 MB exceed typical LLM context windows and are loaded fragmentarily. Prioritize true core content.

Related terms

llms.txt belongs to the operational AI crawler setup - together with robots.txt, GPTBot, Google-Extended and other AI crawler user agents. Strategically it is part of GEO and LLM SEO infrastructure. For documentation sites it complements XML sitemaps as a semantic curation signal.

FAQ on llms.txt