--- title: "llms.txt Guide (2026): How to Create and Optimize the File" url: https://alicelabs.ai/en/insights/llms-txt-guide-2026 description: "llms.txt guide 2026: spec, examples, deployment, validation. Adopted by Anthropic, Stripe, Vercel, Cloudflare, Perplexity, and 800+ domains." lang: en --- Alice Labs (https://alicelabs.ai/en/) Services What we do (https://alicelabs.ai/#welcome)About Alice (https://alicelabs.ai/#who-we-are)Case (https://alicelabs.ai/en/case)Insights (https://alicelabs.ai/en/insights)Contact (https://alicelabs.ai/#email-form) AI Search & LLMO How-to Guide Fresh Last reviewed: 16 September 2026 · 21d ago # llms.txt Guide: How to Create and Optimize the File (2026) ## TL;DR Quick Answer Cited by AI > llms.txt in 2026 is a proposed markdown standard hosted at llmstxt.org, served at yourdomain.com/llms.txt. It gives LLM crawlers a curated H1 title, blockquote summary, and H2 sections of key page links. Anthropic, Stripe, Vercel, Cloudflare, Perplexity, and Hugging Face adopted it by August 2026. llms.txt is a markdown summary of your site placed at the domain root. This guide walks through the format spec, deployment, validation, and ongoing maintenance - with real code you can copy. llms.txt is a proposed web standard introduced by Jeremy Howard at Answer.AI on September 3, 2024. It is a markdown file served at the root of a domain (e.g. yourdomain.com/llms.txt) that gives Large Language Model crawlers a curated, machine-readable summary of the site. The specification lives at llmstxt.org and defines a simple format: an H1 title, a short summary, and lists of links to clean markdown versions of key pages. Time 2-4 hours Difficulty Beginner Tools Any markdown editor (VS Code, Obsidian, plain text), Access to your site root for file deployment, llmstxt.org validators and examples… ## Before you start - Basic familiarity with markdown syntax - Ability to deploy a static file at your domain root - A clear list of your site's most important pages ## What you'll have at the end A valid llms.txt file deployed at your root, optionally paired with llms-full.txt, that gives LLM crawlers a clean, curated summary of your site — and a monitoring loop to track AI crawler activity over time. Image: Linus Ingemarsson - Author at Alice Labs (https://alicelabs.ai/images/linus-ingemarsson.png) Written by Linus Ingemarsson: https://www.linkedin.com/in/linus-ingemarsson/ Image: Eric Lundberg - Reviewer at Alice Labs (https://alicelabs.ai/images/eric-lundberg.png) Reviewed by Eric Lundberg: https://www.linkedin.com/in/eric-lundberg-3530451bb/ Published May 6, 2026 · Updated September 16, 2026 12 min read ## 9-step process 0/9 complete 1. 01 #### Step 1: Inventory the pages an LLM should know about Before writing the file, list the pages you want LLMs to surface: your homepage, product or service pages, top documentation, key blog posts, customer cases, and About/Contact. Skip duplicates, thin pages, archives, and anything time-sensitive. Aim for 20-50 high-signal URLs, not the full sitemap. 2. 02 #### Step 2: Write the H1 title and one-paragraph summary Start the file with a single H1 (the site or product name) and a blockquote summary describing what the site does. Keep the summary to 2-3 sentences. State the entity, the audience, and the value. This block is what LLMs use to classify your site. 3. 03 #### Step 3: Organize the body into H2 sections Group your URLs under H2 headings such as Docs, Blog, Customer Stories, Product, About. Each section should reflect how a human would navigate. Sections give the LLM context about the type of content behind each link, which helps with citation accuracy. 4. 04 #### Step 4: Add links in the spec format Inside each section, list links one per bullet using the format `- [Page Name](url): Optional description`. Keep page names entity-clear. Descriptions should be 5-15 words explaining what is on the page. Always link to canonical URLs, not redirects. 5. 05 #### Step 5: (Optional) Create clean .md versions of key pages The llmstxt.org spec encourages linking to plain markdown versions of pages where possible. LLMs prefer markdown over rendered HTML because it removes nav, ads, and chrome. If your site cannot serve .md, link to the canonical HTML and skip this step — it is optional. 6. 06 #### Step 6: Deploy llms.txt to your site root Save the file as llms.txt and serve it at https://yourdomain.com/llms.txt. The location pattern mirrors robots.txt and sitemap.xml. Make sure the response is 200 OK with Content-Type text/plain or text/markdown. Verify it is reachable in an incognito window. 7. 07 #### Step 7: (Optional) Publish llms-full.txt for full content If your site is small enough, also publish llms-full.txt at the root. This file concatenates the actual clean content of every key page into one document. It is useful for documentation sites and product manuals where the LLM benefits from full context, not just a list of links. 8. 08 #### Step 8: Validate the file using llmstxt.org tools Run your llms.txt through the validators linked from llmstxt.org. Check that markdown parses cleanly, links resolve, and the structure matches the spec. Re-validate every time you ship a content change. Malformed llms.txt is worse than none — it can confuse retrieval. 9. 09 #### Step 9: Monitor AI crawler activity in your logs Once llms.txt is live, watch your server logs for AI crawlers: ClaudeBot, OAI-SearchBot, GPTBot, PerplexityBot, Google-Extended. Note which agents fetch /llms.txt and how often. Combine this with citation tracking (Otterly.ai, Profound) and GA4 referral analytics to measure impact. ## Key Takeaways - llms.txt was proposed by Jeremy Howard (Answer.AI) on September 3, 2024 as a machine-readable site summary for LLM crawlers. - The file is markdown, lives at /llms.txt at your root, and follows a simple H1 + summary + sections + links structure. - It is a convention, not yet officially supported by every major LLM provider — but already adopted by Anthropic, Mintlify, Cloudflare, and many others. - llms.txt does NOT replace robots.txt or sitemap.xml. They coexist: robots.txt controls access, sitemap.xml lists every URL, llms.txt curates the highlights. - The optional llms-full.txt concatenates clean content into one file, useful for smaller sites and documentation. - Pair llms.txt with Schema.org markup, fast-loading clean HTML, and citation-rich content for measurable AI-search visibility (Aggarwal et al. 2024 found up to 40% lift). - Generative AI adoption reached 71% of organizations regularly using it in at least one business function by 2024, up from 65% earlier that year, making machine-readable site summaries a first-mover advantage as of mid-2026 (McKinsey State of AI 2024, https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai). - August 2026 adoption snapshot: 800+ verified domains in the llmstxt.org directory, including Anthropic, Stripe, Vercel, Cloudflare, Perplexity, LangChain, Hugging Face, and Mintlify (auto-generated across 8,000+ hosted docs sites). 01 / 12 Step ## Why llms.txt Exists In short llms.txt was proposed by Jeremy Howard at Answer.AI on September 3, 2024. It exists because LLMs work better with clean markdown than with rendered HTML, and because no native standard existed for sites to summarize themselves for AI crawlers. LLM crawlers face a problem search engines never had to solve. A modern web page is mostly chrome: navigation, ads, cookie banners, JavaScript widgets, and tracking scripts. For a generative model trying to summarize or cite a page, that chrome is noise. Plain markdown — headings, paragraphs, lists — is far easier to process than DOM trees with hundreds of nested divs. llms.txt addresses this gap with a simple convention. A markdown file at your domain root tells an LLM what the site is, what it covers, and where the canonical content lives. The pattern mirrors two existing standards: - **robots.txt** tells crawlers what they may access. - **sitemap.xml** lists every URL on the site. - **llms.txt** curates the high-signal subset and describes it in plain language. llms.txt is a convention. It is not enforced by any LLM provider as of 2026, but Anthropic, Mintlify, Cloudflare, and many others have already published llms.txt files publicly. 02 / 12 Step ## What is llms.txt in 2026? In short llms.txt in 2026 is a proposed markdown standard hosted at llmstxt.org, served at yourdomain.com/llms.txt. It gives LLM crawlers a curated H1 title, blockquote summary, and H2 sections of key page links. Anthropic, Stripe, Vercel, Cloudflare, Perplexity, and Hugging Face adopted it by August 2026. The specification has not changed structurally since Jeremy Howard proposed it on September 3, 2024 — the llmstxt.org repo (github.com/AnswerDotAI/llms-txt) shows only minor clarification PRs in 2025 and 2026: tighter guidance on the`Optional` H2 section, an explicit `llms-full.txt` definition, and content-type recommendations. What changed in 2026 is adoption, not syntax. The file went from ~120 known deployments in early 2025 to 800+ verified in the llmstxt.org directory by August 2026, driven by three things: - Mintlify shipped auto-generation across every docs site it hosts (default-on since Q4 2025). - Cloudflare and Vercel published deployment templates, collapsing setup from hours to minutes. - Anthropic's own `anthropic.com/llms.txt` became a reference implementation cited by dozens of docs teams. For an LLMO strategy, llms.txt sits alongside LLMO consulting (https://alicelabs.ai/en/ai-seo) priorities like Schema.org markup, entity resolution, and citation-worthy content. It is a cheap, fast-to-ship signal, not a citation guarantee. 03 / 12 Step ## Format Spec Deep-Dive (with Examples) In short The llms.txt spec is intentionally minimal: an H1 title, a blockquote summary, optional H2 sections, and lists of links in the format `- [Name](url): description`. Everything is markdown, served as plain text. The full spec is hosted at llmstxt.org. Here is the structure broken down with code. **Minimal valid llms.txt:** ``` # Acme Analytics > Acme Analytics is a B2B product analytics platform for SaaS teams. We help product managers measure feature adoption, retention, and revenue impact in one workspace. ## Docs - [Quickstart](https://acme.example/docs/quickstart): Five-minute setup walkthrough - [API Reference](https://acme.example/docs/api): REST + SDK reference - [Event Schema](https://acme.example/docs/events): Recommended event taxonomy ## Blog - [Product analytics 101](https://acme.example/blog/pa-101): Foundations for new PMs - [Retention math](https://acme.example/blog/retention): Cohort analysis explained ## Customer Stories - [How Linear used Acme to ship faster](https://acme.example/cases/linear) - [Notion's onboarding overhaul](https://acme.example/cases/notion) ## About - [About Acme](https://acme.example/about): Team, mission, funding - [Contact](https://acme.example/contact): Support and sales ``` **Anatomy of the file:** - **Single H1.** The site or product entity name. One line, no decoration. - **Blockquote summary.** A 2-3 sentence elevator pitch. State who it is for, what it does, and the value. - **H2 sections.** Group links by purpose: Docs, Blog, Cases, Product, About. Use whatever sections fit your navigation. - **Bulleted links.** Format: `- [Page Name](url): description`. Description is optional but recommended. Keep total length manageable. A few hundred lines is the sweet spot. The file is meant to be a curated index, not a dump of every URL. 04 / 12 Step ## Where to Deploy and How In short Deploy llms.txt as a static file at your domain root, served at https://yourdomain.com/llms.txt with Content-Type text/plain or text/markdown and an HTTP 200 response. The same pattern as robots.txt and sitemap.xml. Deployment is a static-file problem. There is no backend logic, no database, no auth. Choose the option that matches your stack: - **Static site (Next.js, Astro, Hugo, Jekyll).** Drop llms.txt into your `public/` or `static/` folder and ship. - **WordPress.** Upload to the web root via FTP/SFTP, or use a plugin that exposes a custom file at root. - **CDN (Cloudflare, Vercel, Netlify).** Add a route that serves the file with the correct content type. - **Custom server (Nginx, Apache).** Place llms.txt in your document root next to robots.txt. Verify three things after deploy: 1. **HTTP status:** `curl -I https://yourdomain.com/llms.txt` returns 200. 2. **Content-Type:** `text/plain` or `text/markdown`, not `text/html`. 3. **Public access:** The file loads in an incognito window without auth. llms.txt does not replace robots.txt. Keep both. robots.txt controls crawler access, llms.txt describes the curated content. Image: Linus Ingemarsson (https://alicelabs.ai/images/linus-ingemarsson.png) Image: Eric Lundberg (https://alicelabs.ai/images/eric-lundberg.png) Alice Labs practitioner team ## Talk to the team behind 100+ AI implementations 30-minute discovery call with a senior Alice Labs consultant. No slide deck, no sales pitch — just a scoping conversation. Book a Discovery Call 05 / 12 Step ## Tools That Auto-Generate llms.txt In short Documentation platforms like Mintlify auto-generate llms.txt from your docs source. Static site generators have community plugins that emit the file at build time. For custom sites, a 30-line script can walk your sitemap and produce a draft. You can write llms.txt by hand in under an hour for most sites. For larger sites, automation pays off because the file should stay in sync with your content. Common automation paths: - **Mintlify.** Auto-generates llms.txt and llms-full.txt for documentation sites hosted on Mintlify. - **Static site generator plugins.** Community plugins exist for Next.js, Astro, Hugo, and Docusaurus. They read your content tree and emit llms.txt at build. - **Custom build script.** A small script can read your sitemap.xml, filter by URL pattern, and emit a draft llms.txt. Treat the output as a starting point, then curate by hand. **Example: minimal generator pseudocode** ``` # Pseudocode — adapt to your stack 1. Read sitemap.xml 2. Group URLs by path prefix: /docs/* -> Docs /blog/* -> Blog /cases/* -> Customer Stories 3. Fetch each page's and <meta description> 4. Emit: # {Site name} > {Site summary} ## Docs - [{title}]({url}): {meta description} ... 5. Write to public/llms.txt ``` Whatever route you take, review the output before publishing. An auto-generated llms.txt full of low-quality pages is worse than a hand-curated one with twenty. 06 / 12 Step ## Anatomy of an llms.txt File 2026 In short A 2026-compliant llms.txt has five blocks: H1 entity name, blockquote elevator summary, curated H2 sections with bulleted `- [Name](url): description` links, an optional `## Optional` H2 for lower-priority pages, and a paired `llms-full.txt` if the site is docs-heavy. Every block has a specific job. Here is an annotated example using the pattern Anthropic, Stripe, and Vercel ship in 2026. ``` # Alice Labs # 1. H1 entity name — no tagline, no marketing chrome > Alice Labs is a Nordic AI consultancy that > has delivered 100+ enterprise AI > implementations across Sweden, Denmark, > Norway, and Finland. # 2. Blockquote — 2-3 sentence entity summary ## Docs # 3. Primary H2 section — highest-signal links - [What is LLMO](https://alicelabs.ai/en/insights/what-is-llmo): Glossary definition - [AI Search Optimization Guide](https://alicelabs.ai/en/insights/ai-search-optimization-guide): Pillar guide ## Services - [AI SEO](https://alicelabs.ai/en/ai-seo): LLMO + GEO consulting - [AI Search](https://alicelabs.ai/en/ai-search): Citation benchmarking ## Optional # 4. Optional H2 — lower priority, may be skipped - [About](https://alicelabs.ai/en/about): Team + funding - [Contact](https://alicelabs.ai/en/contact): Support ``` Two 2026 refinements worth calling out: the `## Optional` section has explicit spec support (crawlers may deprioritize it), and Anthropic-style deployments now include an `llms-full.txt`at the same root, concatenating the actual markdown body of every linked page for one-shot ingestion. 07 / 12 Step ## llms.txt vs robots.txt vs sitemap.xml In short The three files coexist and solve different problems. robots.txt gates crawler access. sitemap.xml enumerates every URL for indexing. llms.txt curates a high-signal, human-described subset for LLMs. Publish all three at the domain root. Confusing these three is the most common 2026 mistake we see in audits. Here is the disambiguation. | Attribute | robots.txt | sitemap.xml | llms.txt | | --- | --- | --- | --- | | Format | Plain text directives | XML | Markdown | | Purpose | Gate crawler access | Enumerate every URL | Curate + describe key pages | | Audience | All crawlers | Search engines | LLM crawlers + retrieval | | Size | Tens of lines | Every URL, often 10k+ | 20-50 curated URLs | | Location | /robots.txt | /sitemap.xml | /llms.txt | | Enforced by | Widely honored | Google, Bing, others | Convention (2026) | Practical rule: robots.txt says _what a bot may fetch_, sitemap.xml says _everything that exists_, llms.txt says _the parts an LLM should actually cite_. Alice Labs practitioner team ## Want a hand auditing your AI-search readiness? Alice Labs runs an LLMO Citation Benchmark across ChatGPT, Perplexity, and Claude - including llms.txt, Schema.org, and crawler-access checks - across 100+ Nordic enterprise implementations. Request an LLMO audit 08 / 12 Step ## How LLMs Actually Consume llms.txt In short As of August 2026, no major LLM provider has confirmed inference-time use of llms.txt. Anthropic ships ClaudeBot and publishes its own llms.txt at anthropic.com/llms.txt. OpenAI, Perplexity, and Google discover it via web_search tool calls, not as a first-class ingestion channel. The honest state of consumption in August 2026: - **Anthropic (Claude / ClaudeBot).** Anthropic publishes anthropic.com/llms.txt as a reference. ClaudeBot fetches /llms.txt when it discovers the domain. Anthropic has not documented inference-time use. - **OpenAI (ChatGPT / OAI-SearchBot / GPTBot).** OpenAI documents OAI-SearchBot and GPTBot but has not published guidance for llms.txt. Discovery happens indirectly when web_search retrieves the file. - **Perplexity (PerplexityBot).** Perplexity itself publishes perplexity.ai/llms.txt. Its retrieval engine can surface llms.txt content when it appears in search results. - **Google (Gemini / Google-Extended).** No public statement. Google-Extended controls training data, not llms.txt handling. - **Bing (Copilot).** Bing indexes the file like any other URL. Copilot may cite it via web grounding. Indexability in Bing is a prerequisite for GPT/Copilot citation surface. Treat llms.txt as a discoverable artifact that can enter LLM context via web_search or scheduled crawl, not as a private API to the model. 09 / 12 Step ## Enterprise llms.txt Patterns In short Enterprise llms.txt patterns in 2026 cover four content types: product docs (quickstart + API reference + SDK), changelog (dated releases as bullets), API reference (versioned endpoints), and legal (terms, privacy, DPA). Split by audience or ship a single curated file capped at 50 links. The four enterprise patterns we ship for Nordic clients doing AI SEO consulting (https://alicelabs.ai/en/ai-seo) engagements: - **Product docs.** H2 sections: Quickstart, Core concepts, Guides, API reference, SDK. This is the Stripe and Vercel pattern. Links go to markdown-friendly doc pages, not marketing. - **Changelog.** Bulleted, dated release notes at a stable URL, referenced from llms.txt. LLMs use dates to resolve freshness questions ("what changed in v3.2?"). - **API reference.** Versioned endpoints linked individually. Stripe's llms.txt exposes each API resource (Charges, Customers, Subscriptions) as its own bullet. - **Terms, privacy, DPA.** A `## Legal` section listing terms of service, privacy policy, and Data Processing Agreement. This is what LLM answer engines cite for compliance questions. Cap the total at ~50 links. Beyond that, an LLM cannot use every entry and the signal-to-noise drops. 10 / 12 Step ## How to Generate llms.txt Automatically In short Three 2026 auto-generation paths: Mintlify auto-generates for hosted docs, Cloudflare Workers can emit llms.txt from a KV-stored URL list at the edge, and static generator scripts (Node, Python) walk sitemap.xml + fetch metadata to produce a draft on every build. Which path fits which stack: - **Mintlify hosted docs.** llms.txt and llms-full.txt are auto-generated. No config required as of Q4 2025. Verify the output; edit source docs to control it. - **Cloudflare Workers.** A Worker route reads a curated URL list from Workers KV, formats it as markdown, and returns `text/plain`. Setup takes ~30 minutes. Cloudflare published a template in 2026. - **Vercel edge function.** Same pattern in Next.js: a route handler at `app/llms.txt/route.ts` reads a curated JSON list and emits markdown. Deploys on every push. - **Static script (Node / Python).** Read sitemap.xml, group URLs by prefix, fetch each page's`<title>` and meta description, emit a draft to `public/llms.txt`. Curate the output by hand before shipping. The generation code is trivial; the curation is the work. Never publish an auto-generated file untouched. Alice Labs practitioner team Book a Discovery Call 11 / 12 Step ## Common llms.txt Mistakes to Avoid In short The six most common 2026 mistakes: dumping the full sitemap instead of curating, vague link names like `[Pricing]`, missing H1 or blockquote summary, blocking AI crawlers in robots.txt while publishing llms.txt, hosting on a subpath instead of the root, and never refreshing after content changes. The mistakes we catch most often in audits: 1. **Sitemap dump.** Copying every URL from sitemap.xml into llms.txt destroys the signal. Cap at 20-50 curated links. 2. **Vague link text.** `[Pricing]` is worthless when extracted in isolation. Write `[Alice Labs Pricing]`. 3. **Missing H1 or blockquote.** The spec requires both. Validators at llmstxt.org reject files without them. 4. **Contradicting robots.txt.** Publishing llms.txt while blocking ClaudeBot, OAI-SearchBot, or PerplexityBot in robots.txt is self-defeating. 5. **Wrong location.** llms.txt must be at `yourdomain.com/llms.txt`. Not `/docs/llms.txt`, not on a subdomain unless the subdomain is a separate LLMs-facing product. 6. **Stale file.** A file last touched at launch becomes misleading fast. Set a 90-day review cycle. See the maintenance loop below. 12 / 12 Step ## Validation and Ongoing Maintenance In short Validate llms.txt with the tools at llmstxt.org after every change. Set a 90-day review cycle to refresh links, descriptions, and the summary. Watch server logs for AI crawler activity to confirm the file is being fetched. llms.txt is not fire-and-forget. Your site changes — new pages, renamed URLs, retired products. The file must keep up. Establish three loops: 1. **Validation on every change.** Run llms.txt through the validators at llmstxt.org. Confirm markdown parses, links resolve, and the structure matches the spec. 2. **90-day content review.** Re-check every link. Update descriptions for pages that have evolved. Add new pages that earned their place. Remove pages that no longer matter. 3. **Crawler-log monitoring.** Filter server logs by AI user agents (ClaudeBot, OAI-SearchBot, GPTBot, PerplexityBot, Google-Extended). Track which fetch `/llms.txt` and how often. Pair these loops with downstream measurement. Track AI citations with prompt audits or tools like Otterly.ai and Profound. Watch GA4 referral traffic from chatgpt.com, perplexity.ai, and claude.ai. llms.txt by itself does not produce citations. It works as one input in a larger LLMO system that includes Schema.org markup, entity-clear writing, fresh dates, and authoritative sourcing. For a curated shortlist of vendors that operationalize these patterns, see our ai content optimization tools (https://alicelabs.ai/en/insights/ai-content-optimization-tools) listicle. ## About the Authors & Reviewers Written by Image: Linus Ingemarsson - CEO & Co-Founder, Alice Labs at Alice Labs (https://alicelabs.ai/images/linus-ingemarsson.png) Linus Ingemarsson (https://www.linkedin.com/in/linus-ingemarsson/) CEO & Co-Founder, Alice Labs CEO & Co-Founder at Alice Labs. Author of 7 research reports on AI adoption, governance and labor markets cited across EU, OECD and US benchmarks. - 8+ years in AI strategy & implementation - Top-5 AI Speaker, Sweden (Mindley 2025) - 100+ enterprise AI engagements View profile: https://www.linkedin.com/in/linus-ingemarsson/ Linus Ingemarsson on LinkedIn: https://www.linkedin.com/in/linus-ingemarsson/ Email Linus Ingemarsson: linus@alicelabs.ai Reviewed by September 16, 2026 Image: Eric Lundberg - Co-Founder, Alice Labs at Alice Labs (https://alicelabs.ai/images/eric-lundberg.png) Eric Lundberg (https://www.linkedin.com/in/eric-lundberg-3530451bb/) Co-Founder, Alice Labs Co-Founder at Alice Labs. Builds AI automation, agent workflows and integration systems that hold up in real business operations. - AI automation & agent systems lead - Workflow design across 100+ deployments - Specialist in RAG, integrations & APIs View profile: https://www.linkedin.com/in/eric-lundberg-3530451bb/ Eric Lundberg on LinkedIn: https://www.linkedin.com/in/eric-lundberg-3530451bb/ Email Eric Lundberg: eric@alicelabs.ai Reviewed for technical accuracy, methodology and source integrity. · All claims trace to public sources cited in-line. ## Frequently Asked Questions ### Is llms.txt an official standard? No. llms.txt is a proposed convention published by Jeremy Howard at Answer.AI on September 3, 2024 at llmstxt.org. It is not yet ratified by any standards body, and no major LLM provider has publicly committed to honoring it during inference. It is widely adopted in practice by sites including Anthropic, Mintlify, and Cloudflare. ### Does Google support llms.txt? Google has not publicly confirmed that it uses llms.txt for AI Overviews or Gemini grounding. Google does, however, respect Schema.org structured data and the Google-Extended user agent. Treat llms.txt as a complement to those signals, not a replacement. ### Does Claude or ChatGPT actually read llms.txt? Anthropic publishes its own llms.txt file, but neither Anthropic nor OpenAI has publicly stated that ClaudeBot or OAI-SearchBot use llms.txt during inference or retrieval. The convention is widely adopted; verified inference-time use is not guaranteed. The upside is asymmetric: low cost, zero downside. ### Should I publish llms-full.txt as well? Yes if your site is small or documentation-focused. llms-full.txt concatenates the actual content of your key pages into one markdown file, so an LLM can ingest the full text without crawling individual URLs. For larger sites, the linked llms.txt alone is usually enough. ### Where exactly do I put the llms.txt file? At the root of your domain, served at https://yourdomain.com/llms.txt. The location mirrors robots.txt and sitemap.xml. Make sure the response is HTTP 200, Content-Type text/plain or text/markdown, and accessible without authentication. ### Does llms.txt replace robots.txt or sitemap.xml? No. The three files coexist. robots.txt controls which crawlers may access which paths. sitemap.xml lists every URL for search engines. llms.txt curates a high-signal subset for LLM crawlers and describes it in markdown. You should publish all three. ### What are common mistakes when creating llms.txt? The most common mistakes are: dumping the entire sitemap instead of curating, using vague link names like `Pricing` instead of `Acme Pricing`, omitting the H1 or summary, blocking AI crawlers in robots.txt while publishing llms.txt, and never updating the file after content changes. Run through the llmstxt.org validators after every edit to catch structural issues. ### What is the llms.txt specification format in 2025 and 2026? The llms.txt specification, maintained at llmstxt.org since September 3, 2024, has not changed structurally in 2025 or 2026. The format remains: a single H1 with the site or product name, a blockquote of 2-3 sentence summary, then H2 sections listing links as `- [Page Name](url): description`. An optional `## Optional` section may hold lower-priority links. Serve the file at the root as text/plain or text/markdown. ### Does llms.txt go on the site root or somewhere else? llms.txt must go at the domain root, served at https://yourdomain.com/llms.txt — the exact same location pattern as robots.txt and sitemap.xml. Subdirectories, subdomains for docs, or CDN paths are not valid locations per the llmstxt.org spec. Crawlers only look at the root. The optional llms-full.txt also lives at the root (https://yourdomain.com/llms-full.txt), never inside /docs/ or /content/. ### How often should I update llms.txt? Update llms.txt whenever you ship a major content change — new product pages, renamed URLs, retired sections. Set a 90-day review cycle to re-check links, refresh descriptions, and prune dead entries. Re-validate at llmstxt.org after every change. ### What is llms.txt in one sentence? llms.txt is a markdown file served at yourdomain.com/llms.txt that gives LLM crawlers a curated H1 title, blockquote summary, and H2 sections of key page links — a low-cost signal of AI-readiness proposed by Jeremy Howard (Answer.AI) in September 2024 and adopted by 800+ domains by August 2026. ### How is llms.txt different from robots.txt? robots.txt gates crawler access with allow/disallow directives in plain text. llms.txt curates a small set of high-signal pages for LLMs and describes them in markdown. robots.txt says what a bot may fetch; llms.txt says what an LLM should cite. They coexist; publish both at the domain root. ### Which LLMs actually read llms.txt? As of August 2026, Anthropic publishes anthropic.com/llms.txt and ClaudeBot fetches /llms.txt when it discovers a domain, but Anthropic has not confirmed inference-time use. OpenAI, Perplexity, and Google discover llms.txt indirectly via web_search tool calls. Perplexity itself publishes perplexity.ai/llms.txt. Treat consumption as web-search-mediated, not private API. ### Do I actually need llms.txt for my site? If you want LLM visibility, yes — the cost is a single markdown file and the downside is zero. It becomes higher-leverage for documentation, API references, product catalogs, and content sites where LLMs benefit from a curated map. Pure brochureware with fewer than ten pages benefits less; the signal is already discoverable. ### How do I write llms.txt from scratch? Start with a single H1 (your entity name), a blockquote 2-3 sentence summary, then H2 sections (Docs, Blog, Cases) with bulleted `- [Page Name](url): description` links. Cap at 20-50 curated URLs. Save as llms.txt at your web root. Serve as text/plain or text/markdown with HTTP 200. Validate at llmstxt.org. ### Does llms.txt help with LLM citations? It is one input among many. No LLM provider guarantees citation lift from llms.txt alone. Aggarwal et al. (2024) found citation-rich content lifts generative-engine visibility up to 40%; llms.txt makes that content easier for crawlers to find. Pair it with Schema.org markup, entity resolution, and clean markdown for compounding impact. ### Can I generate llms.txt automatically? Yes. Three 2026 paths: Mintlify auto-generates for hosted docs (default-on since Q4 2025); Cloudflare Workers and Vercel edge functions can emit llms.txt from a curated URL list at request time; and a static Node or Python script can walk sitemap.xml and produce a draft at build time. Always review the output before publishing. ### Does OpenAI read llms.txt? OpenAI has not published guidance for llms.txt. OAI-SearchBot and GPTBot fetch the file if they discover it, and ChatGPT's web_search tool can retrieve it during grounding, but there is no documented inference-time use. Being indexable in Bing is the prerequisite: no Bing presence, no ChatGPT or Copilot citation surface — llms.txt cannot fix that. ### What are the most common llms.txt mistakes to avoid? Dumping the full sitemap instead of curating; vague link text like `[Pricing]` instead of `[Alice Labs Pricing]`; missing H1 or blockquote summary; blocking ClaudeBot or OAI-SearchBot in robots.txt while publishing llms.txt; hosting at /docs/llms.txt instead of the root; and shipping the file once then never refreshing. Run the llmstxt.org validators after every edit. Previous in AI Search & LLMO ### Zero-Click Search in the AI Era: What Marketers Need to Know https://alicelabs.ai/en/insights/zero-click-search-ai-era Next in AI Search & LLMO ### How to Get Cited by Claude: How Claude Finds Sources (2026) https://alicelabs.ai/en/insights/how-to-get-cited-by-claude ## Further reading - llmstxt.org — Official spec, directory, and validators (https://llmstxt.org/)· llmstxt.org - Anthropic — anthropic.com/llms.txt (reference implementation) (https://www.anthropic.com/llms.txt)· anthropic.com - Stripe — stripe.com/llms.txt (enterprise docs pattern) (https://stripe.com/llms.txt)· stripe.com - Vercel — vercel.com/llms.txt (deployment template) (https://vercel.com/llms.txt)· vercel.com - Cloudflare — cloudflare.com/llms.txt (edge implementation) (https://cloudflare.com/llms.txt)· cloudflare.com - Perplexity — perplexity.ai/llms.txt (LLM provider deployment) (https://perplexity.ai/llms.txt)· perplexity.ai - GEO: Generative Engine Optimization (Aggarwal et al., 2024) (https://arxiv.org/abs/2311.09735)· arxiv.org - Search Engine Land — LLMO coverage library (https://searchengineland.com/library/generative-ai)· searchengineland.com ## Related reading glossary ### What Is LLMO? Large Language Model Optimization Explained Glossary definition of the discipline llms.txt fits within. 7 min https://alicelabs.ai/en/insights/what-is-llmo pillar ### AI Search Optimization: Complete Guide for 2026 Full playbook covering ChatGPT, Perplexity, Claude, and Google AI Overviews. 14 min https://alicelabs.ai/en/insights/ai-search-optimization-guide howto ### How to Get Cited by ChatGPT: 9-Step Playbook Tactical step-by-step playbook for ChatGPT citation, where llms.txt is one of nine factors. 11 min https://alicelabs.ai/en/insights/how-to-get-cited-by-chatgpt ## Sources 1. llms.txt — Official specification, Jeremy Howard / Answer.AI (proposed September 3, 2024, spec repo AnswerDotAI/llms-txt on GitHub) (https://llmstxt.org/)(accessed 2026-08-14) 2. Aggarwal et al. — GEO: Generative Engine Optimization (arXiv:2311.09735, 2024) (https://arxiv.org/abs/2311.09735)(accessed 2026-08-14) 3. Anthropic — anthropic.com/llms.txt (reference implementation) + Claude docs on ClaudeBot user agent (https://docs.anthropic.com/en/docs/claude-code/overview)(accessed 2026-08-14) 4. Vercel — Publishing llms.txt on Vercel (deployment template, 2026) (https://vercel.com/llms.txt)(accessed 2026-08-14) 5. Cloudflare — llms.txt with Cloudflare Workers (edge implementation blog) (https://cloudflare.com/llms.txt)(accessed 2026-08-14) 6. Stripe — stripe.com/llms.txt (docs-first enterprise reference) (https://stripe.com/llms.txt)(accessed 2026-08-14) 7. Perplexity — perplexity.ai/llms.txt (LLM provider self-deployment) (https://perplexity.ai/llms.txt)(accessed 2026-08-14) 8. Search Engine Land — LLMO and llms.txt adoption coverage (2026) (https://searchengineland.com/library/generative-ai)(accessed 2026-08-14) 9. OpenAI — Robots.txt and OAI-SearchBot / GPTBot documentation (https://platform.openai.com/docs/bots)(accessed 2026-08-14) 10. Schema.org — Article, FAQPage, HowTo type documentation (https://schema.org/Article)(accessed 2026-08-14) Next scheduled review: 2026-12-15 Alice Labs practitioner team Book a Discovery Call Share Share on LinkedIn (https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Falicelabs.ai%2Fen%2Finsights%2Fllms-txt-guide-2026)Share on X (https://twitter.com/intent/tweet?url=https%3A%2F%2Falicelabs.ai%2Fen%2Finsights%2Fllms-txt-guide-2026&text=llms.txt%20Guide%20(2026)%3A%20How%20to%20Create%20and%20Optimize%20the%20File) ## Get in Touch! The lab usually responds within 24 hours. Need help with AI? Get in touch (https://alicelabs.ai/en/ai-consulting-services#contact-form) ## Structured data ```json [ { "@context": "https://schema.org", "@graph": [ { "@type": "Organization", "@id": "https://alicelabs.ai/#organization", "name": "Alice Labs", "alternateName": [ "Alice Labs AB", "AliceLabs" ], "legalName": "Alice Labs AB", "identifier": "559443-5470", "foundingLocation": { "@type": "Place", "name": "Stockholm, Sweden" }, "url": "https://alicelabs.ai", "logo": { "@type": "ImageObject", "@id": "https://alicelabs.ai/#logo", "url": "https://alicelabs.ai/images/alice-logo.png", "contentUrl": "https://alicelabs.ai/images/alice-logo.png", "width": 2000, "height": 2027, "caption": "Alice Labs" }, "image": { "@id": "https://alicelabs.ai/#logo" }, "description": "Alice Labs är en svensk AI-byrå som hjälper företag implementera AI - från strategi till skalning.", "slogan": "From AI strategy to measurable results.", "foundingDate": "2023", "email": "hej@alicelabs.ai", "telephone": "+46734157476", "address": { "@type": "PostalAddress", "streetAddress": "Hammarbybacken 27", "addressLocality": "Stockholm", "postalCode": "120 30", "addressCountry": "SE" }, "contactPoint": [ { "@type": "ContactPoint", "contactType": "customer service", "email": "hej@alicelabs.ai", "telephone": "+46734157476", "areaServed": [ "SE", "EU" ], "availableLanguage": [ "Swedish", "English" ] } ], "areaServed": [ { "@type": "Country", "name": "Sweden" }, { "@type": "Place", "name": "Europe" } ], "knowsAbout": [ "AI strategy", "AI implementation", "AI agents", "AI automation", "Generative AI", "AI governance", "AI training", "Machine learning", "Large language models", "RAG", "AI consulting", "Digital transformation", "AI search optimization", "LLMO", "AI for enterprise" ], "founder": [ { "@id": "https://alicelabs.ai/#linus" }, { "@id": "https://alicelabs.ai/#eric" } ], "sameAs": [ "https://www.linkedin.com/company/alicelabsai", "https://www.trustpilot.com/review/alicelabs.ai", "https://www.wikidata.org/wiki/Q140369570" ] }, { "@type": "Person", "@id": "https://alicelabs.ai/#linus", "name": "Linus Ingemarsson", "givenName": "Linus", "familyName": "Ingemarsson", "jobTitle": "Co-Founder", "description": "Co-founder of Alice Labs. Architects AI agent systems and automation in production for clients across financial services, media, and the public sector.", "url": "https://alicelabs.ai/en/linus-ingemarsson", "sameAs": [ "https://www.linkedin.com/in/linus-ingemarsson/", "https://www.wikidata.org/wiki/Q140369914" ], "knowsAbout": [ "AI agents", "agent orchestration", "AI implementation", "LangGraph", "RAG systems", "AI strategy", "enterprise AI", "AI search optimization", "LLMO", "Nordic AI ecosystem" ], "worksFor": { "@id": "https://alicelabs.ai/#organization" } }, { "@type": "Person", "@id": "https://alicelabs.ai/#eric", "name": "Eric Lundberg", "givenName": "Eric", "familyName": "Lundberg", "jobTitle": "Co-Founder", "description": "Co-founder of Alice Labs. Designs AI automation systems and agent workflows that remove repetitive work and make day-to-day operations more reliable.", "url": "https://alicelabs.ai/en/eric-lundberg", "sameAs": [ "https://www.linkedin.com/in/eric-lundberg-3530451bb/", "https://www.wikidata.org/wiki/Q140369978" ], "knowsAbout": [ "AI automation", "agent workflows", "AI integrations", "process automation", "knowledge systems", "AI engineering", "enterprise AI", "Nordic AI ecosystem" ], "worksFor": { "@id": "https://alicelabs.ai/#organization" } }, { "@type": "Person", "@id": "https://alicelabs.ai/#alice", "name": "Alice Holmgren", "givenName": "Alice", "familyName": "Holmgren", "jobTitle": "CEO", "description": "CEO of Alice Labs. Leads strategy and growth across the Nordic AI consulting market.", "url": "https://alicelabs.ai/en/alice-holmgren", "knowsAbout": [ "AI strategy", "AI consulting leadership", "business development", "Nordic AI ecosystem", "enterprise AI adoption", "AI program management" ], "worksFor": { "@id": "https://alicelabs.ai/#organization" } }, { "@type": [ "LocalBusiness", "ProfessionalService" ], "@id": "https://alicelabs.ai/#localbusiness", "name": "Alice Labs", "description": "AI-konsult i Stockholm. Vi hjälper företag implementera AI - från strategi till skalning. Boka möte för en kostnadsfri AI-genomgång.", "url": "https://alicelabs.ai", "logo": { "@id": "https://alicelabs.ai/#logo" }, "image": { "@id": "https://alicelabs.ai/#logo" }, "telephone": "+46734157476", "email": "hej@alicelabs.ai", "priceRange": "$$$", "currenciesAccepted": "SEK, EUR, USD", "paymentAccepted": "Invoice", "address": { "@type": "PostalAddress", "streetAddress": "Hammarbybacken 27", "addressLocality": "Stockholm", "postalCode": "120 30", "addressRegion": "Stockholms län", "addressCountry": "SE" }, "geo": { "@type": "GeoCoordinates", "latitude": 59.3018, "longitude": 18.1003 }, "areaServed": [ { "@type": "City", "name": "Stockholm" }, { "@type": "City", "name": "Göteborg" }, { "@type": "City", "name": "Malmö" }, { "@type": "City", "name": "Uppsala" }, { "@type": "Country", "name": "Sweden" } ], "openingHoursSpecification": [ { "@type": "OpeningHoursSpecification", "dayOfWeek": [ "Monday", "Tuesday", "Wednesday", "Thursday", "Friday" ], "opens": "08:00", "closes": "18:00" } ], "hasOfferCatalog": { "@type": "OfferCatalog", "name": "AI-tjänster", "itemListElement": [ { "@type": "Offer", "itemOffered": { "@type": "Service", "name": "AI-konsult" } }, { "@type": "Offer", "itemOffered": { "@type": "Service", "name": "AI-strategi" } }, { "@type": "Offer", "itemOffered": { "@type": "Service", "name": "AI-implementation" } }, { "@type": "Offer", "itemOffered": { "@type": "Service", "name": "AI-utbildning" } }, { "@type": "Offer", "itemOffered": { "@type": "Service", "name": "AI-agenter" } }, { "@type": "Offer", "itemOffered": { "@type": "Service", "name": "AI-automation" } } ] }, "knowsAbout": [ "AI-konsult", "AI-strategi", "AI-implementation", "AI-utbildning", "AI-agenter", "AI-automation", "Generative AI", "Machine learning", "RAG", "Large language models", "AI governance" ], "parentOrganization": { "@id": "https://alicelabs.ai/#organization" }, "sameAs": [ "https://www.linkedin.com/company/alicelabsai" ] }, { "@type": "WebSite", "@id": "https://alicelabs.ai/#website", "url": "https://alicelabs.ai", "name": "Alice Labs", "alternateName": [ "Alice Labs AB" ], "description": "AI consulting, implementation and training for businesses.", "publisher": { "@id": "https://alicelabs.ai/#organization" }, "inLanguage": [ "sv-SE", "en-US" ], "potentialAction": { "@type": "SearchAction", "target": { "@type": "EntryPoint", "urlTemplate": "https://alicelabs.ai/?q={search_term_string}" }, "query-input": "required name=search_term_string" } } ] }, { "@context": "https://schema.org", "@graph": [ { "@type": "HowTo", "@id": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#article", "headline": "llms.txt Guide: How to Create and Optimize the File (2026)", "description": "llms.txt guide 2026: spec, examples, deployment, validation. Adopted by Anthropic, Stripe, Vercel, Cloudflare, Perplexity, and 800+ domains.", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026", "datePublished": "2026-05-06", "dateModified": "2026-09-16", "expires": "2026-12-15", "author": { "@id": "https://alicelabs.ai/#linus" }, "reviewedBy": { "@id": "https://alicelabs.ai/#eric" }, "dateReviewed": "2026-09-16", "publisher": { "@type": "Organization", "name": "Alice Labs", "url": "https://alicelabs.ai", "logo": { "@type": "ImageObject", "url": "https://alicelabs.ai/images/alice-logo.png" } }, "image": { "@type": "ImageObject", "@id": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#hero-image", "url": "https://alicelabs.ai/images/og/og-home.jpg", "contentUrl": "https://alicelabs.ai/images/og/og-home.jpg", "width": 1600, "height": 900, "caption": "llms.txt Guide (2026): How to Create and Optimize the File", "creator": { "@id": "https://alicelabs.ai/#organization" }, "representativeOfPage": true, "license": "https://alicelabs.ai/terms" }, "mainEntityOfPage": { "@type": "WebPage", "@id": "https://alicelabs.ai/en/insights/llms-txt-guide-2026" }, "inLanguage": "en", "articleSection": "ai-search", "keywords": "llms.txt guide, llms.txt setup, llms.txt example, ai crawler txt file, how to create llms.txt, llms-full.txt, llmstxt.org", "about": [ { "@type": "Thing", "name": "Why llms.txt Exists", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#why-llms-txt-exists" }, { "@type": "Thing", "name": "What is llms.txt in 2026?", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#what-is-llms-txt-2026" }, { "@type": "Thing", "name": "Format Spec Deep-Dive (with Examples)", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#format-spec-deep-dive" }, { "@type": "Thing", "name": "Where to Deploy and How", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#where-to-deploy" }, { "@type": "Thing", "name": "Tools That Auto-Generate llms.txt", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#tools-that-auto-generate" }, { "@type": "Thing", "name": "Anatomy of an llms.txt File 2026", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#anatomy-2026" }, { "@type": "Thing", "name": "llms.txt vs robots.txt vs sitemap.xml", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#vs-robots-sitemap" }, { "@type": "Thing", "name": "How LLMs Actually Consume llms.txt", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#how-llms-consume" }, { "@type": "Thing", "name": "Enterprise llms.txt Patterns", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#enterprise-patterns" }, { "@type": "Thing", "name": "How to Generate llms.txt Automatically", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#auto-generate" }, { "@type": "Thing", "name": "Common llms.txt Mistakes to Avoid", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#common-mistakes" }, { "@type": "Thing", "name": "Validation and Ongoing Maintenance", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#validation-and-maintenance" } ], "mentions": [ { "@type": "Organization", "name": "Alice Labs", "url": "https://alicelabs.ai" }, { "@type": "Organization", "name": "Google", "url": "https://google.com" }, { "@type": "Organization", "name": "OpenAI", "url": "https://openai.com" }, { "@type": "Organization", "name": "Anthropic", "url": "https://anthropic.com" }, { "@type": "Organization", "name": "European Union", "url": "https://europa.eu" }, { "@type": "Product", "name": "ChatGPT", "url": "https://chatgpt.com" }, { "@type": "Product", "name": "Claude", "url": "https://claude.ai" }, { "@type": "Product", "name": "Google Gemini", "url": "https://gemini.google.com" }, { "@type": "Person", "name": "Eric Lundberg", "url": "https://linkedin.com/in/eric-lundberg-3530451bb" }, { "@type": "Person", "name": "Linus Ingemarsson", "url": "https://linkedin.com/in/linus-ingemarsson" }, { "@type": "Organization", "name": "Stripe", "url": "https://stripe.com" }, { "@type": "Organization", "name": "Vercel", "url": "https://vercel.com" }, { "@type": "Organization", "name": "Cloudflare", "url": "https://cloudflare.com" }, { "@type": "Organization", "name": "Perplexity", "url": "https://perplexity.ai" }, { "@type": "Organization", "name": "Hugging Face", "url": "https://huggingface.co" }, { "@type": "Organization", "name": "LangChain", "url": "https://langchain.com" }, { "@type": "Organization", "name": "Mintlify", "url": "https://mintlify.com" } ], "hasPart": [ { "@type": "WebPageElement", "isAccessibleForFree": true, "name": "Why llms.txt Exists", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#why-llms-txt-exists", "description": "llms.txt was proposed by Jeremy Howard at Answer.AI on September 3, 2024. It exists because LLMs work better with clean markdown than with rendered HTML, and because no native standard existed for sites to summarize themselves for AI crawlers." }, { "@type": "WebPageElement", "isAccessibleForFree": true, "name": "What is llms.txt in 2026?", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#what-is-llms-txt-2026", "description": "llms.txt in 2026 is a proposed markdown standard hosted at llmstxt.org, served at yourdomain.com/llms.txt. It gives LLM crawlers a curated H1 title, blockquote summary, and H2 sections of key page links. Anthropic, Stripe, Vercel, Cloudflare, Perplexity, and Hugging Face adopted it by August 2026." }, { "@type": "WebPageElement", "isAccessibleForFree": true, "name": "Format Spec Deep-Dive (with Examples)", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#format-spec-deep-dive", "description": "The llms.txt spec is intentionally minimal: an H1 title, a blockquote summary, optional H2 sections, and lists of links in the format `- [Name](url): description`. Everything is markdown, served as plain text." }, { "@type": "WebPageElement", "isAccessibleForFree": true, "name": "Where to Deploy and How", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#where-to-deploy", "description": "Deploy llms.txt as a static file at your domain root, served at https://yourdomain.com/llms.txt with Content-Type text/plain or text/markdown and an HTTP 200 response. The same pattern as robots.txt and sitemap.xml." }, { "@type": "WebPageElement", "isAccessibleForFree": true, "name": "Tools That Auto-Generate llms.txt", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#tools-that-auto-generate", "description": "Documentation platforms like Mintlify auto-generate llms.txt from your docs source. Static site generators have community plugins that emit the file at build time. For custom sites, a 30-line script can walk your sitemap and produce a draft." }, { "@type": "WebPageElement", "isAccessibleForFree": true, "name": "Anatomy of an llms.txt File 2026", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#anatomy-2026", "description": "A 2026-compliant llms.txt has five blocks: H1 entity name, blockquote elevator summary, curated H2 sections with bulleted `- [Name](url): description` links, an optional `## Optional` H2 for lower-priority pages, and a paired `llms-full.txt` if the site is docs-heavy." }, { "@type": "WebPageElement", "isAccessibleForFree": true, "name": "llms.txt vs robots.txt vs sitemap.xml", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#vs-robots-sitemap", "description": "The three files coexist and solve different problems. robots.txt gates crawler access. sitemap.xml enumerates every URL for indexing. llms.txt curates a high-signal, human-described subset for LLMs. Publish all three at the domain root." }, { "@type": "WebPageElement", "isAccessibleForFree": true, "name": "How LLMs Actually Consume llms.txt", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#how-llms-consume", "description": "As of August 2026, no major LLM provider has confirmed inference-time use of llms.txt. Anthropic ships ClaudeBot and publishes its own llms.txt at anthropic.com/llms.txt. OpenAI, Perplexity, and Google discover it via web_search tool calls, not as a first-class ingestion channel." }, { "@type": "WebPageElement", "isAccessibleForFree": true, "name": "Enterprise llms.txt Patterns", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#enterprise-patterns", "description": "Enterprise llms.txt patterns in 2026 cover four content types: product docs (quickstart + API reference + SDK), changelog (dated releases as bullets), API reference (versioned endpoints), and legal (terms, privacy, DPA). Split by audience or ship a single curated file capped at 50 links." }, { "@type": "WebPageElement", "isAccessibleForFree": true, "name": "How to Generate llms.txt Automatically", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#auto-generate", "description": "Three 2026 auto-generation paths: Mintlify auto-generates for hosted docs, Cloudflare Workers can emit llms.txt from a KV-stored URL list at the edge, and static generator scripts (Node, Python) walk sitemap.xml + fetch metadata to produce a draft on every build." }, { "@type": "WebPageElement", "isAccessibleForFree": true, "name": "Common llms.txt Mistakes to Avoid", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#common-mistakes", "description": "The six most common 2026 mistakes: dumping the full sitemap instead of curating, vague link names like `[Pricing]`, missing H1 or blockquote summary, blocking AI crawlers in robots.txt while publishing llms.txt, hosting on a subpath instead of the root, and never refreshing after content changes." }, { "@type": "WebPageElement", "isAccessibleForFree": true, "name": "Validation and Ongoing Maintenance", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#validation-and-maintenance", "description": "Validate llms.txt with the tools at llmstxt.org after every change. Set a 90-day review cycle to refresh links, descriptions, and the summary. Watch server logs for AI crawler activity to confirm the file is being fetched." } ], "speakable": { "@type": "SpeakableSpecification", "cssSelector": [ "[data-speakable='true']", "[data-snippet='true']", "[data-section-answer='true']", ".quick-answer", "h1" ] } }, { "@type": "BreadcrumbList", "@id": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#breadcrumb", "itemListElement": [ { "@type": "ListItem", "position": 1, "name": "Home", "item": "https://alicelabs.ai/en" }, { "@type": "ListItem", "position": 2, "name": "Insights", "item": "https://alicelabs.ai/en/insights" }, { "@type": "ListItem", "position": 3, "name": "ai-search", "item": "https://alicelabs.ai/en/insights/ai-search" }, { "@type": "ListItem", "position": 4, "name": "llms.txt Guide (2026): How to Create and Optimize the File", "item": "https://alicelabs.ai/en/insights/llms-txt-guide-2026" } ] }, { "@type": "Person", "@id": "https://alicelabs.ai/#eric", "name": "Eric Lundberg", "jobTitle": "Co-Founder", "worksFor": { "@id": "https://alicelabs.ai/#organization" }, "knowsAbout": [ { "@type": "DefinedTerm", "name": "AI automation", "url": "https://www.wikidata.org/wiki/Q1322483" }, { "@type": "DefinedTerm", "name": "Workflow automation", "url": "https://www.wikidata.org/wiki/Q120427660" }, { "@type": "DefinedTerm", "name": "Retrieval-Augmented Generation", "url": "https://www.wikidata.org/wiki/Q117761563" }, { "@type": "DefinedTerm", "name": "Enterprise AI implementation" } ], "sameAs": [ "https://www.linkedin.com/in/eric-lundberg-3530451bb/", "https://www.wikidata.org/wiki/Q140369978" ] }, { "@type": "Person", "@id": "https://alicelabs.ai/#linus", "name": "Linus Ingemarsson", "jobTitle": "CEO & Co-Founder", "worksFor": { "@id": "https://alicelabs.ai/#organization" }, "knowsAbout": [ { "@type": "DefinedTerm", "name": "AI agent orchestration", "url": "https://www.wikidata.org/wiki/Q98678395" }, { "@type": "DefinedTerm", "name": "AI strategy" }, { "@type": "DefinedTerm", "name": "AI search optimization (LLMO)" }, { "@type": "DefinedTerm", "name": "Enterprise AI strategy" }, { "@type": "DefinedTerm", "name": "AI consulting leadership" } ], "sameAs": [ "https://www.linkedin.com/in/linus-ingemarsson/", "https://www.wikidata.org/wiki/Q140369914" ] }, { "@type": "FAQPage", "mainEntity": [ { "@type": "Question", "name": "Is llms.txt an official standard?", "acceptedAnswer": { "@type": "Answer", "text": "No. llms.txt is a proposed convention published by Jeremy Howard at Answer.AI on September 3, 2024 at llmstxt.org. It is not yet ratified by any standards body, and no major LLM provider has publicly committed to honoring it during inference. It is widely adopted in practice by sites including Anthropic, Mintlify, and Cloudflare." } }, { "@type": "Question", "name": "Does Google support llms.txt?", "acceptedAnswer": { "@type": "Answer", "text": "Google has not publicly confirmed that it uses llms.txt for AI Overviews or Gemini grounding. Google does, however, respect Schema.org structured data and the Google-Extended user agent. Treat llms.txt as a complement to those signals, not a replacement." } }, { "@type": "Question", "name": "Does Claude or ChatGPT actually read llms.txt?", "acceptedAnswer": { "@type": "Answer", "text": "Anthropic publishes its own llms.txt file, but neither Anthropic nor OpenAI has publicly stated that ClaudeBot or OAI-SearchBot use llms.txt during inference or retrieval. The convention is widely adopted; verified inference-time use is not guaranteed. The upside is asymmetric: low cost, zero downside." } }, { "@type": "Question", "name": "Should I publish llms-full.txt as well?", "acceptedAnswer": { "@type": "Answer", "text": "Yes if your site is small or documentation-focused. llms-full.txt concatenates the actual content of your key pages into one markdown file, so an LLM can ingest the full text without crawling individual URLs. For larger sites, the linked llms.txt alone is usually enough." } }, { "@type": "Question", "name": "Where exactly do I put the llms.txt file?", "acceptedAnswer": { "@type": "Answer", "text": "At the root of your domain, served at https://yourdomain.com/llms.txt. The location mirrors robots.txt and sitemap.xml. Make sure the response is HTTP 200, Content-Type text/plain or text/markdown, and accessible without authentication." } }, { "@type": "Question", "name": "Does llms.txt replace robots.txt or sitemap.xml?", "acceptedAnswer": { "@type": "Answer", "text": "No. The three files coexist. robots.txt controls which crawlers may access which paths. sitemap.xml lists every URL for search engines. llms.txt curates a high-signal subset for LLM crawlers and describes it in markdown. You should publish all three." } }, { "@type": "Question", "name": "What are common mistakes when creating llms.txt?", "acceptedAnswer": { "@type": "Answer", "text": "The most common mistakes are: dumping the entire sitemap instead of curating, using vague link names like `Pricing` instead of `Acme Pricing`, omitting the H1 or summary, blocking AI crawlers in robots.txt while publishing llms.txt, and never updating the file after content changes. Run through the llmstxt.org validators after every edit to catch structural issues." } }, { "@type": "Question", "name": "What is the llms.txt specification format in 2025 and 2026?", "acceptedAnswer": { "@type": "Answer", "text": "The llms.txt specification, maintained at llmstxt.org since September 3, 2024, has not changed structurally in 2025 or 2026. The format remains: a single H1 with the site or product name, a blockquote of 2-3 sentence summary, then H2 sections listing links as `- [Page Name](url): description`. An optional `## Optional` section may hold lower-priority links. Serve the file at the root as text/plain or text/markdown." } }, { "@type": "Question", "name": "Does llms.txt go on the site root or somewhere else?", "acceptedAnswer": { "@type": "Answer", "text": "llms.txt must go at the domain root, served at https://yourdomain.com/llms.txt — the exact same location pattern as robots.txt and sitemap.xml. Subdirectories, subdomains for docs, or CDN paths are not valid locations per the llmstxt.org spec. Crawlers only look at the root. The optional llms-full.txt also lives at the root (https://yourdomain.com/llms-full.txt), never inside /docs/ or /content/." } }, { "@type": "Question", "name": "How often should I update llms.txt?", "acceptedAnswer": { "@type": "Answer", "text": "Update llms.txt whenever you ship a major content change — new product pages, renamed URLs, retired sections. Set a 90-day review cycle to re-check links, refresh descriptions, and prune dead entries. Re-validate at llmstxt.org after every change." } }, { "@type": "Question", "name": "What is llms.txt in one sentence?", "acceptedAnswer": { "@type": "Answer", "text": "llms.txt is a markdown file served at yourdomain.com/llms.txt that gives LLM crawlers a curated H1 title, blockquote summary, and H2 sections of key page links — a low-cost signal of AI-readiness proposed by Jeremy Howard (Answer.AI) in September 2024 and adopted by 800+ domains by August 2026." } }, { "@type": "Question", "name": "How is llms.txt different from robots.txt?", "acceptedAnswer": { "@type": "Answer", "text": "robots.txt gates crawler access with allow/disallow directives in plain text. llms.txt curates a small set of high-signal pages for LLMs and describes them in markdown. robots.txt says what a bot may fetch; llms.txt says what an LLM should cite. They coexist; publish both at the domain root." } }, { "@type": "Question", "name": "Which LLMs actually read llms.txt?", "acceptedAnswer": { "@type": "Answer", "text": "As of August 2026, Anthropic publishes anthropic.com/llms.txt and ClaudeBot fetches /llms.txt when it discovers a domain, but Anthropic has not confirmed inference-time use. OpenAI, Perplexity, and Google discover llms.txt indirectly via web_search tool calls. Perplexity itself publishes perplexity.ai/llms.txt. Treat consumption as web-search-mediated, not private API." } }, { "@type": "Question", "name": "Do I actually need llms.txt for my site?", "acceptedAnswer": { "@type": "Answer", "text": "If you want LLM visibility, yes — the cost is a single markdown file and the downside is zero. It becomes higher-leverage for documentation, API references, product catalogs, and content sites where LLMs benefit from a curated map. Pure brochureware with fewer than ten pages benefits less; the signal is already discoverable." } }, { "@type": "Question", "name": "How do I write llms.txt from scratch?", "acceptedAnswer": { "@type": "Answer", "text": "Start with a single H1 (your entity name), a blockquote 2-3 sentence summary, then H2 sections (Docs, Blog, Cases) with bulleted `- [Page Name](url): description` links. Cap at 20-50 curated URLs. Save as llms.txt at your web root. Serve as text/plain or text/markdown with HTTP 200. Validate at llmstxt.org." } }, { "@type": "Question", "name": "Does llms.txt help with LLM citations?", "acceptedAnswer": { "@type": "Answer", "text": "It is one input among many. No LLM provider guarantees citation lift from llms.txt alone. Aggarwal et al. (2024) found citation-rich content lifts generative-engine visibility up to 40%; llms.txt makes that content easier for crawlers to find. Pair it with Schema.org markup, entity resolution, and clean markdown for compounding impact." } }, { "@type": "Question", "name": "Can I generate llms.txt automatically?", "acceptedAnswer": { "@type": "Answer", "text": "Yes. Three 2026 paths: Mintlify auto-generates for hosted docs (default-on since Q4 2025); Cloudflare Workers and Vercel edge functions can emit llms.txt from a curated URL list at request time; and a static Node or Python script can walk sitemap.xml and produce a draft at build time. Always review the output before publishing." } }, { "@type": "Question", "name": "Does OpenAI read llms.txt?", "acceptedAnswer": { "@type": "Answer", "text": "OpenAI has not published guidance for llms.txt. OAI-SearchBot and GPTBot fetch the file if they discover it, and ChatGPT's web_search tool can retrieve it during grounding, but there is no documented inference-time use. Being indexable in Bing is the prerequisite: no Bing presence, no ChatGPT or Copilot citation surface — llms.txt cannot fix that." } }, { "@type": "Question", "name": "What are the most common llms.txt mistakes to avoid?", "acceptedAnswer": { "@type": "Answer", "text": "Dumping the full sitemap instead of curating; vague link text like `[Pricing]` instead of `[Alice Labs Pricing]`; missing H1 or blockquote summary; blocking ClaudeBot or OAI-SearchBot in robots.txt while publishing llms.txt; hosting at /docs/llms.txt instead of the root; and shipping the file once then never refreshing. Run the llmstxt.org validators after every edit." } } ] }, { "@context": "https://schema.org", "@type": "HowTo", "name": "llms.txt Guide: How to Create and Optimize the File (2026)", "description": "llms.txt guide 2026: spec, examples, deployment, validation. Adopted by Anthropic, Stripe, Vercel, Cloudflare, Perplexity, and 800+ domains.", "totalTime": "12 min read", "step": [ { "@type": "HowToStep", "position": 1, "name": "Inventory the pages an LLM should know about", "text": "Before writing the file, list the pages you want LLMs to surface: your homepage, product or service pages, top documentation, key blog posts, customer cases, and About/Contact. Skip duplicates, thin pages, archives, and anything time-sensitive. Aim for 20-50 high-signal URLs, not the full sitemap." }, { "@type": "HowToStep", "position": 2, "name": "Write the H1 title and one-paragraph summary", "text": "Start the file with a single H1 (the site or product name) and a blockquote summary describing what the site does. Keep the summary to 2-3 sentences. State the entity, the audience, and the value. This block is what LLMs use to classify your site." }, { "@type": "HowToStep", "position": 3, "name": "Organize the body into H2 sections", "text": "Group your URLs under H2 headings such as Docs, Blog, Customer Stories, Product, About. Each section should reflect how a human would navigate. Sections give the LLM context about the type of content behind each link, which helps with citation accuracy." }, { "@type": "HowToStep", "position": 4, "name": "Add links in the spec format", "text": "Inside each section, list links one per bullet using the format `- [Page Name](url): Optional description`. Keep page names entity-clear. Descriptions should be 5-15 words explaining what is on the page. Always link to canonical URLs, not redirects." }, { "@type": "HowToStep", "position": 5, "name": "(Optional) Create clean .md versions of key pages", "text": "The llmstxt.org spec encourages linking to plain markdown versions of pages where possible. LLMs prefer markdown over rendered HTML because it removes nav, ads, and chrome. If your site cannot serve .md, link to the canonical HTML and skip this step — it is optional." }, { "@type": "HowToStep", "position": 6, "name": "Deploy llms.txt to your site root", "text": "Save the file as llms.txt and serve it at https://yourdomain.com/llms.txt. The location pattern mirrors robots.txt and sitemap.xml. Make sure the response is 200 OK with Content-Type text/plain or text/markdown. Verify it is reachable in an incognito window." }, { "@type": "HowToStep", "position": 7, "name": "(Optional) Publish llms-full.txt for full content", "text": "If your site is small enough, also publish llms-full.txt at the root. This file concatenates the actual clean content of every key page into one document. It is useful for documentation sites and product manuals where the LLM benefits from full context, not just a list of links." }, { "@type": "HowToStep", "position": 8, "name": "Validate the file using llmstxt.org tools", "text": "Run your llms.txt through the validators linked from llmstxt.org. Check that markdown parses cleanly, links resolve, and the structure matches the spec. Re-validate every time you ship a content change. Malformed llms.txt is worse than none — it can confuse retrieval." }, { "@type": "HowToStep", "position": 9, "name": "Monitor AI crawler activity in your logs", "text": "Once llms.txt is live, watch your server logs for AI crawlers: ClaudeBot, OAI-SearchBot, GPTBot, PerplexityBot, Google-Extended. Note which agents fetch /llms.txt and how often. Combine this with citation tracking (Otterly.ai, Profound) and GA4 referral analytics to measure impact." } ] }, { "@context": "https://schema.org", "@type": "Dataset", "name": "llms.txt Guide: How to Create and Optimize the File (2026)", "description": "llms.txt guide 2026: spec, examples, deployment, validation. Adopted by Anthropic, Stripe, Vercel, Cloudflare, Perplexity, and 800+ domains.", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026", "datePublished": "2026-05-06", "dateModified": "2026-09-16", "creator": { "@type": "Organization", "name": "Alice Labs", "url": "https://alicelabs.ai" }, "license": "https://creativecommons.org/licenses/by/4.0/", "isAccessibleForFree": true, "keywords": [ "llms.txt guide", "llms.txt setup", "llms.txt example", "ai crawler txt file", "how to create llms.txt", "llms-full.txt", "llmstxt.org" ] }, { "@context": "https://schema.org", "@type": "ItemList", "name": "Related articles", "itemListElement": [ { "@type": "ListItem", "position": 1, "url": "https://alicelabs.ai/en/insights/what-is-llmo", "name": "What Is LLMO? Large Language Model Optimization Explained" }, { "@type": "ListItem", "position": 2, "url": "https://alicelabs.ai/en/insights/ai-search-optimization-guide", "name": "AI Search Optimization: Complete Guide for 2026" }, { "@type": "ListItem", "position": 3, "url": "https://alicelabs.ai/en/insights/how-to-get-cited-by-chatgpt", "name": "How to Get Cited by ChatGPT: 9-Step Playbook" } ] }, { "@context": "https://schema.org", "@type": "ItemList", "name": "Table of Contents", "numberOfItems": 12, "itemListOrder": "https://schema.org/ItemListOrderAscending", "itemListElement": [ { "@type": "ListItem", "position": 1, "name": "Why llms.txt Exists", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#why-llms-txt-exists" }, { "@type": "ListItem", "position": 2, "name": "What is llms.txt in 2026?", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#what-is-llms-txt-2026" }, { "@type": "ListItem", "position": 3, "name": "Format Spec Deep-Dive (with Examples)", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#format-spec-deep-dive" }, { "@type": "ListItem", "position": 4, "name": "Where to Deploy and How", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#where-to-deploy" }, { "@type": "ListItem", "position": 5, "name": "Tools That Auto-Generate llms.txt", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#tools-that-auto-generate" }, { "@type": "ListItem", "position": 6, "name": "Anatomy of an llms.txt File 2026", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#anatomy-2026" }, { "@type": "ListItem", "position": 7, "name": "llms.txt vs robots.txt vs sitemap.xml", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#vs-robots-sitemap" }, { "@type": "ListItem", "position": 8, "name": "How LLMs Actually Consume llms.txt", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#how-llms-consume" }, { "@type": "ListItem", "position": 9, "name": "Enterprise llms.txt Patterns", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#enterprise-patterns" }, { "@type": "ListItem", "position": 10, "name": "How to Generate llms.txt Automatically", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#auto-generate" }, { "@type": "ListItem", "position": 11, "name": "Common llms.txt Mistakes to Avoid", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#common-mistakes" }, { "@type": "ListItem", "position": 12, "name": "Validation and Ongoing Maintenance", "url": "https://alicelabs.ai/en/insights/llms-txt-guide-2026#validation-and-maintenance" } ] } ] }, { "@context": "https://schema.org", "@type": "BreadcrumbList", "itemListElement": [ { "@type": "ListItem", "position": 1, "name": "Home", "item": "https://alicelabs.ai/en" }, { "@type": "ListItem", "position": 2, "name": "Insights", "item": "https://alicelabs.ai/en/insights" }, { "@type": "ListItem", "position": 3, "name": "AI Search & LLMO", "item": "https://alicelabs.ai/en/insights/ai-search" }, { "@type": "ListItem", "position": 4, "name": "llms.txt Guide: How to Create and Optimize the File (2026)" } ] } ] ```