llms.txt Guide: How to Create and Optimize the File (2026)

Why llms.txt Exists

In short

llms.txt was proposed by Jeremy Howard at Answer.AI on September 3, 2024. It exists because LLMs work better with clean markdown than with rendered HTML, and because no native standard existed for sites to summarize themselves for AI crawlers.

LLM crawlers face a problem search engines never had to solve. A modern web page is mostly chrome: navigation, ads, cookie banners, JavaScript widgets, and tracking scripts.

For a generative model trying to summarize or cite a page, that chrome is noise. Plain markdown — headings, paragraphs, lists — is far easier to process than DOM trees with hundreds of nested divs.

llms.txt addresses this gap with a simple convention. A markdown file at your domain root tells an LLM what the site is, what it covers, and where the canonical content lives.

The pattern mirrors two existing standards:

• robots.txt tells crawlers what they may access.
• sitemap.xml lists every URL on the site.
• llms.txt curates the high-signal subset and describes it in plain language.

llms.txt is a convention. It is not enforced by any LLM provider as of 2026, but Anthropic, Mintlify, Cloudflare, and many others have already published llms.txt files publicly.

Format Spec Deep-Dive (with Examples)

In short

The llms.txt spec is intentionally minimal: an H1 title, a blockquote summary, optional H2 sections, and lists of links in the format `- [Name](url): description`. Everything is markdown, served as plain text.

The full spec is hosted at llmstxt.org. Here is the structure broken down with code.

Minimal valid llms.txt:

# Acme Analytics

> Acme Analytics is a B2B product analytics platform for SaaS teams. We help product managers measure feature adoption, retention, and revenue impact in one workspace.

## Docs

- [Quickstart](https://acme.example/docs/quickstart): Five-minute setup walkthrough
- [API Reference](https://acme.example/docs/api): REST + SDK reference
- [Event Schema](https://acme.example/docs/events): Recommended event taxonomy

## Blog

- [Product analytics 101](https://acme.example/blog/pa-101): Foundations for new PMs
- [Retention math](https://acme.example/blog/retention): Cohort analysis explained

## Customer Stories

- [How Linear used Acme to ship faster](https://acme.example/cases/linear)
- [Notion's onboarding overhaul](https://acme.example/cases/notion)

## About

- [About Acme](https://acme.example/about): Team, mission, funding
- [Contact](https://acme.example/contact): Support and sales

Anatomy of the file:

• Single H1. The site or product entity name. One line, no decoration.
• Blockquote summary. A 2-3 sentence elevator pitch. State who it is for, what it does, and the value.
• H2 sections. Group links by purpose: Docs, Blog, Cases, Product, About. Use whatever sections fit your navigation.
• Bulleted links. Format: - [Page Name](url): description. Description is optional but recommended.

Keep total length manageable. A few hundred lines is the sweet spot. The file is meant to be a curated index, not a dump of every URL.

Where to Deploy and How

In short

Deploy llms.txt as a static file at your domain root, served at https://yourdomain.com/llms.txt with Content-Type text/plain or text/markdown and an HTTP 200 response. The same pattern as robots.txt and sitemap.xml.

Deployment is a static-file problem. There is no backend logic, no database, no auth.

Choose the option that matches your stack:

• Static site (Next.js, Astro, Hugo, Jekyll). Drop llms.txt into your public/ or static/ folder and ship.
• WordPress. Upload to the web root via FTP/SFTP, or use a plugin that exposes a custom file at root.
• CDN (Cloudflare, Vercel, Netlify). Add a route that serves the file with the correct content type.
• Custom server (Nginx, Apache). Place llms.txt in your document root next to robots.txt.

Verify three things after deploy:

1. HTTP status: curl -I https://yourdomain.com/llms.txt returns 200.
2. Content-Type: text/plain or text/markdown, not text/html.
3. Public access: The file loads in an incognito window without auth.

llms.txt does not replace robots.txt. Keep both. robots.txt controls crawler access, llms.txt describes the curated content.

Tools That Auto-Generate llms.txt

In short

Documentation platforms like Mintlify auto-generate llms.txt from your docs source. Static site generators have community plugins that emit the file at build time. For custom sites, a 30-line script can walk your sitemap and produce a draft.

You can write llms.txt by hand in under an hour for most sites. For larger sites, automation pays off because the file should stay in sync with your content.

Common automation paths:

• Mintlify. Auto-generates llms.txt and llms-full.txt for documentation sites hosted on Mintlify.
• Static site generator plugins. Community plugins exist for Next.js, Astro, Hugo, and Docusaurus. They read your content tree and emit llms.txt at build.
• Custom build script. A small script can read your sitemap.xml, filter by URL pattern, and emit a draft llms.txt. Treat the output as a starting point, then curate by hand.

Example: minimal generator pseudocode

# Pseudocode — adapt to your stack
1. Read sitemap.xml
2. Group URLs by path prefix:
   /docs/* -> Docs
   /blog/* -> Blog
   /cases/* -> Customer Stories
3. Fetch each page's <title> and <meta description>
4. Emit:
   # {Site name}
   > {Site summary}

   ## Docs
   - [{title}]({url}): {meta description}
   ...
5. Write to public/llms.txt

Whatever route you take, review the output before publishing. An auto-generated llms.txt full of low-quality pages is worse than a hand-curated one with twenty.

Validation and Ongoing Maintenance

In short

Validate llms.txt with the tools at llmstxt.org after every change. Set a 90-day review cycle to refresh links, descriptions, and the summary. Watch server logs for AI crawler activity to confirm the file is being fetched.

llms.txt is not fire-and-forget. Your site changes — new pages, renamed URLs, retired products. The file must keep up.

Establish three loops:

1. Validation on every change. Run llms.txt through the validators at llmstxt.org. Confirm markdown parses, links resolve, and the structure matches the spec.
2. 90-day content review. Re-check every link. Update descriptions for pages that have evolved. Add new pages that earned their place. Remove pages that no longer matter.
3. Crawler-log monitoring. Filter server logs by AI user agents (ClaudeBot, OAI-SearchBot, GPTBot, PerplexityBot, Google-Extended). Track which fetch /llms.txt and how often.

Pair these loops with downstream measurement. Track AI citations with prompt audits or tools like Otterly.ai and Profound. Watch GA4 referral traffic from chatgpt.com, perplexity.ai, and claude.ai.

llms.txt by itself does not produce citations. It works as one input in a larger LLMO system that includes Schema.org markup, entity-clear writing, fresh dates, and authoritative sourcing.