llms.txt Guide: How to Create and Optimize the File (2026)

Why llms.txt Exists

In short

llms.txt was proposed by Jeremy Howard at Answer.AI on September 3, 2024. It exists because LLMs work better with clean markdown than with rendered HTML, and because no native standard existed for sites to summarize themselves for AI crawlers.

LLM crawlers face a problem search engines never had to solve. A modern web page is mostly chrome: navigation, ads, cookie banners, JavaScript widgets, and tracking scripts.

For a generative model trying to summarize or cite a page, that chrome is noise. Plain markdown — headings, paragraphs, lists — is far easier to process than DOM trees with hundreds of nested divs.

llms.txt addresses this gap with a simple convention. A markdown file at your domain root tells an LLM what the site is, what it covers, and where the canonical content lives.

The pattern mirrors two existing standards:

• robots.txt tells crawlers what they may access.
• sitemap.xml lists every URL on the site.
• llms.txt curates the high-signal subset and describes it in plain language.

llms.txt is a convention. It is not enforced by any LLM provider as of 2026, but Anthropic, Mintlify, Cloudflare, and many others have already published llms.txt files publicly.

What is llms.txt in 2026?

In short

llms.txt in 2026 is a proposed markdown standard hosted at llmstxt.org, served at yourdomain.com/llms.txt. It gives LLM crawlers a curated H1 title, blockquote summary, and H2 sections of key page links. Anthropic, Stripe, Vercel, Cloudflare, Perplexity, and Hugging Face adopted it by August 2026.

The specification has not changed structurally since Jeremy Howard proposed it on September 3, 2024 — the llmstxt.org repo (github.com/AnswerDotAI/llms-txt) shows only minor clarification PRs in 2025 and 2026: tighter guidance on theOptional H2 section, an explicit llms-full.txt definition, and content-type recommendations.

What changed in 2026 is adoption, not syntax. The file went from ~120 known deployments in early 2025 to 800+ verified in the llmstxt.org directory by August 2026, driven by three things:

• Mintlify shipped auto-generation across every docs site it hosts (default-on since Q4 2025).
• Cloudflare and Vercel published deployment templates, collapsing setup from hours to minutes.
• Anthropic's own anthropic.com/llms.txt became a reference implementation cited by dozens of docs teams.

For an LLMO strategy, llms.txt sits alongside LLMO consulting priorities like Schema.org markup, entity resolution, and citation-worthy content. It is a cheap, fast-to-ship signal, not a citation guarantee.

Format Spec Deep-Dive (with Examples)

In short

The llms.txt spec is intentionally minimal: an H1 title, a blockquote summary, optional H2 sections, and lists of links in the format `- [Name](url): description`. Everything is markdown, served as plain text.

The full spec is hosted at llmstxt.org. Here is the structure broken down with code.

Minimal valid llms.txt:

# Acme Analytics

> Acme Analytics is a B2B product analytics platform for SaaS teams. We help product managers measure feature adoption, retention, and revenue impact in one workspace.

## Docs

- [Quickstart](https://acme.example/docs/quickstart): Five-minute setup walkthrough
- [API Reference](https://acme.example/docs/api): REST + SDK reference
- [Event Schema](https://acme.example/docs/events): Recommended event taxonomy

## Blog

- [Product analytics 101](https://acme.example/blog/pa-101): Foundations for new PMs
- [Retention math](https://acme.example/blog/retention): Cohort analysis explained

## Customer Stories

- [How Linear used Acme to ship faster](https://acme.example/cases/linear)
- [Notion's onboarding overhaul](https://acme.example/cases/notion)

## About

- [About Acme](https://acme.example/about): Team, mission, funding
- [Contact](https://acme.example/contact): Support and sales

Anatomy of the file:

• Single H1. The site or product entity name. One line, no decoration.
• Blockquote summary. A 2-3 sentence elevator pitch. State who it is for, what it does, and the value.
• H2 sections. Group links by purpose: Docs, Blog, Cases, Product, About. Use whatever sections fit your navigation.
• Bulleted links. Format: - [Page Name](url): description. Description is optional but recommended.

Keep total length manageable. A few hundred lines is the sweet spot. The file is meant to be a curated index, not a dump of every URL.

Where to Deploy and How

In short

Deploy llms.txt as a static file at your domain root, served at https://yourdomain.com/llms.txt with Content-Type text/plain or text/markdown and an HTTP 200 response. The same pattern as robots.txt and sitemap.xml.

Deployment is a static-file problem. There is no backend logic, no database, no auth.

Choose the option that matches your stack:

• Static site (Next.js, Astro, Hugo, Jekyll). Drop llms.txt into your public/ or static/ folder and ship.
• WordPress. Upload to the web root via FTP/SFTP, or use a plugin that exposes a custom file at root.
• CDN (Cloudflare, Vercel, Netlify). Add a route that serves the file with the correct content type.
• Custom server (Nginx, Apache). Place llms.txt in your document root next to robots.txt.

Verify three things after deploy:

1. HTTP status: curl -I https://yourdomain.com/llms.txt returns 200.
2. Content-Type: text/plain or text/markdown, not text/html.
3. Public access: The file loads in an incognito window without auth.

llms.txt does not replace robots.txt. Keep both. robots.txt controls crawler access, llms.txt describes the curated content.

Tools That Auto-Generate llms.txt

In short

Documentation platforms like Mintlify auto-generate llms.txt from your docs source. Static site generators have community plugins that emit the file at build time. For custom sites, a 30-line script can walk your sitemap and produce a draft.

You can write llms.txt by hand in under an hour for most sites. For larger sites, automation pays off because the file should stay in sync with your content.

Common automation paths:

• Mintlify. Auto-generates llms.txt and llms-full.txt for documentation sites hosted on Mintlify.
• Static site generator plugins. Community plugins exist for Next.js, Astro, Hugo, and Docusaurus. They read your content tree and emit llms.txt at build.
• Custom build script. A small script can read your sitemap.xml, filter by URL pattern, and emit a draft llms.txt. Treat the output as a starting point, then curate by hand.

Example: minimal generator pseudocode

# Pseudocode — adapt to your stack
1. Read sitemap.xml
2. Group URLs by path prefix:
   /docs/* -> Docs
   /blog/* -> Blog
   /cases/* -> Customer Stories
3. Fetch each page's <title> and <meta description>
4. Emit:
   # {Site name}
   > {Site summary}

   ## Docs
   - [{title}]({url}): {meta description}
   ...
5. Write to public/llms.txt

Whatever route you take, review the output before publishing. An auto-generated llms.txt full of low-quality pages is worse than a hand-curated one with twenty.

Anatomy of an llms.txt File 2026

In short

A 2026-compliant llms.txt has five blocks: H1 entity name, blockquote elevator summary, curated H2 sections with bulleted `- [Name](url): description` links, an optional `## Optional` H2 for lower-priority pages, and a paired `llms-full.txt` if the site is docs-heavy.

Every block has a specific job. Here is an annotated example using the pattern Anthropic, Stripe, and Vercel ship in 2026.

# Alice Labs                              # 1. H1 entity name — no tagline, no marketing chrome

> Alice Labs is a Nordic AI consultancy that
> has delivered 100+ enterprise AI
> implementations across Sweden, Denmark,
> Norway, and Finland.                     # 2. Blockquote — 2-3 sentence entity summary

## Docs                                    # 3. Primary H2 section — highest-signal links
- [What is LLMO](https://alicelabs.ai/en/insights/what-is-llmo): Glossary definition
- [AI Search Optimization Guide](https://alicelabs.ai/en/insights/ai-search-optimization-guide): Pillar guide

## Services
- [AI SEO](https://alicelabs.ai/en/ai-seo): LLMO + GEO consulting
- [AI Search](https://alicelabs.ai/en/ai-search): Citation benchmarking

## Optional                                # 4. Optional H2 — lower priority, may be skipped
- [About](https://alicelabs.ai/en/about): Team + funding
- [Contact](https://alicelabs.ai/en/contact): Support

Two 2026 refinements worth calling out: the ## Optional section has explicit spec support (crawlers may deprioritize it), and Anthropic-style deployments now include an llms-full.txtat the same root, concatenating the actual markdown body of every linked page for one-shot ingestion.

llms.txt vs robots.txt vs sitemap.xml

In short

The three files coexist and solve different problems. robots.txt gates crawler access. sitemap.xml enumerates every URL for indexing. llms.txt curates a high-signal, human-described subset for LLMs. Publish all three at the domain root.

Confusing these three is the most common 2026 mistake we see in audits. Here is the disambiguation.

Practical rule: robots.txt says what a bot may fetch, sitemap.xml says everything that exists, llms.txt says the parts an LLM should actually cite.

How LLMs Actually Consume llms.txt

In short

As of August 2026, no major LLM provider has confirmed inference-time use of llms.txt. Anthropic ships ClaudeBot and publishes its own llms.txt at anthropic.com/llms.txt. OpenAI, Perplexity, and Google discover it via web_search tool calls, not as a first-class ingestion channel.

The honest state of consumption in August 2026:

• Anthropic (Claude / ClaudeBot). Anthropic publishes anthropic.com/llms.txt as a reference. ClaudeBot fetches /llms.txt when it discovers the domain. Anthropic has not documented inference-time use.
• OpenAI (ChatGPT / OAI-SearchBot / GPTBot). OpenAI documents OAI-SearchBot and GPTBot but has not published guidance for llms.txt. Discovery happens indirectly when web_search retrieves the file.
• Perplexity (PerplexityBot). Perplexity itself publishes perplexity.ai/llms.txt. Its retrieval engine can surface llms.txt content when it appears in search results.
• Google (Gemini / Google-Extended). No public statement. Google-Extended controls training data, not llms.txt handling.
• Bing (Copilot). Bing indexes the file like any other URL. Copilot may cite it via web grounding. Indexability in Bing is a prerequisite for GPT/Copilot citation surface.

Treat llms.txt as a discoverable artifact that can enter LLM context via web_search or scheduled crawl, not as a private API to the model.

Enterprise llms.txt Patterns

In short

Enterprise llms.txt patterns in 2026 cover four content types: product docs (quickstart + API reference + SDK), changelog (dated releases as bullets), API reference (versioned endpoints), and legal (terms, privacy, DPA). Split by audience or ship a single curated file capped at 50 links.

The four enterprise patterns we ship for Nordic clients doing AI SEO consulting engagements:

• Product docs. H2 sections: Quickstart, Core concepts, Guides, API reference, SDK. This is the Stripe and Vercel pattern. Links go to markdown-friendly doc pages, not marketing.
• Changelog. Bulleted, dated release notes at a stable URL, referenced from llms.txt. LLMs use dates to resolve freshness questions ("what changed in v3.2?").
• API reference. Versioned endpoints linked individually. Stripe's llms.txt exposes each API resource (Charges, Customers, Subscriptions) as its own bullet.
• Terms, privacy, DPA. A ## Legal section listing terms of service, privacy policy, and Data Processing Agreement. This is what LLM answer engines cite for compliance questions.

Cap the total at ~50 links. Beyond that, an LLM cannot use every entry and the signal-to-noise drops.

How to Generate llms.txt Automatically

In short

Three 2026 auto-generation paths: Mintlify auto-generates for hosted docs, Cloudflare Workers can emit llms.txt from a KV-stored URL list at the edge, and static generator scripts (Node, Python) walk sitemap.xml + fetch metadata to produce a draft on every build.

Which path fits which stack:

• Mintlify hosted docs. llms.txt and llms-full.txt are auto-generated. No config required as of Q4 2025. Verify the output; edit source docs to control it.
• Cloudflare Workers. A Worker route reads a curated URL list from Workers KV, formats it as markdown, and returns text/plain. Setup takes ~30 minutes. Cloudflare published a template in 2026.
• Vercel edge function. Same pattern in Next.js: a route handler at app/llms.txt/route.ts reads a curated JSON list and emits markdown. Deploys on every push.
• Static script (Node / Python). Read sitemap.xml, group URLs by prefix, fetch each page's<title> and meta description, emit a draft to public/llms.txt. Curate the output by hand before shipping.

The generation code is trivial; the curation is the work. Never publish an auto-generated file untouched.

Common llms.txt Mistakes to Avoid

In short

The six most common 2026 mistakes: dumping the full sitemap instead of curating, vague link names like `[Pricing]`, missing H1 or blockquote summary, blocking AI crawlers in robots.txt while publishing llms.txt, hosting on a subpath instead of the root, and never refreshing after content changes.

The mistakes we catch most often in audits:

1. Sitemap dump. Copying every URL from sitemap.xml into llms.txt destroys the signal. Cap at 20-50 curated links.
2. Vague link text. [Pricing] is worthless when extracted in isolation. Write [Alice Labs Pricing].
3. Missing H1 or blockquote. The spec requires both. Validators at llmstxt.org reject files without them.
4. Contradicting robots.txt. Publishing llms.txt while blocking ClaudeBot, OAI-SearchBot, or PerplexityBot in robots.txt is self-defeating.
5. Wrong location. llms.txt must be at yourdomain.com/llms.txt. Not /docs/llms.txt, not on a subdomain unless the subdomain is a separate LLMs-facing product.
6. Stale file. A file last touched at launch becomes misleading fast. Set a 90-day review cycle. See the maintenance loop below.

Validation and Ongoing Maintenance

In short

Validate llms.txt with the tools at llmstxt.org after every change. Set a 90-day review cycle to refresh links, descriptions, and the summary. Watch server logs for AI crawler activity to confirm the file is being fetched.

llms.txt is not fire-and-forget. Your site changes — new pages, renamed URLs, retired products. The file must keep up.

Establish three loops:

1. Validation on every change. Run llms.txt through the validators at llmstxt.org. Confirm markdown parses, links resolve, and the structure matches the spec.
2. 90-day content review. Re-check every link. Update descriptions for pages that have evolved. Add new pages that earned their place. Remove pages that no longer matter.
3. Crawler-log monitoring. Filter server logs by AI user agents (ClaudeBot, OAI-SearchBot, GPTBot, PerplexityBot, Google-Extended). Track which fetch /llms.txt and how often.

Pair these loops with downstream measurement. Track AI citations with prompt audits or tools like Otterly.ai and Profound. Watch GA4 referral traffic from chatgpt.com, perplexity.ai, and claude.ai.

llms.txt by itself does not produce citations. It works as one input in a larger LLMO system that includes Schema.org markup, entity-clear writing, fresh dates, and authoritative sourcing.

For a curated shortlist of vendors that operationalize these patterns, see our ai content optimization tools listicle.