Site Architecture for AI Crawlers: A 2026 Engineering Guide

Why Architecture Matters for AI Crawlers (the Topical Authority Signal)

In short

AI crawlers reward sites that are obviously organized around topics. A hub-and-spoke architecture — pillar page plus 5-15 supporting articles, all densely interlinked — is the clearest signal of topical authority an LLM can extract from a domain.

For two decades, SEO architecture meant siloing content under categories so PageRank could flow cleanly. The mechanics were link-based.

AI crawlers see the same structures but interpret them differently. What they extract is not link equity but topical breadth and depth — how many related entities a domain covers, and how tightly those entities are connected.

Topical authority for LLMs is the product of three things: breadth of coverage across a topic, depth on each subtopic, and the density of internal links that show how subtopics relate.

What changed in the last 24 months.

• Citations replaced clicks. SparkToro's 2024 study found roughly 60% of searches end zero-click. Visibility now means being cited inside an answer, not ranked above it.
• LLM crawlers traverse, then summarize. Bots like ClaudeBot and OAI-SearchBot follow internal links to understand context, then condense what they find into citation-ready passages.
• Architecture became extractable. BreadcrumbList schema, llms.txt, and sitemap.xml all expose hierarchy in machine-readable form. Sites that publish this structure are cited preferentially.

Aggarwal et al. (2024) found citation-rich, well-structured content sees up to 40% improved visibility in generative engines. The structure is not decorative — it is part of the signal.

Site architecture is the foundation under everything else in LLMO. Schema, llms.txt, and citation-quality writing all assume the underlying topology is clean.

The Hub-and-Spoke Pattern (with Diagram)

In short

Hub-and-spoke is the dominant content architecture for AI search. One pillar page covers the broad topic; 5-15 spoke articles each go deep on one subtopic. Every spoke links back to the hub, every hub links forward to its spokes, and adjacent spokes link to each other.

The pattern has three roles: the pillar (hub), the supporting articles (spokes), and the contextual links that connect them.

Visualizing the structure.

                    [ Pillar — broad topic ]
                              |
        +---------+-----------+-----------+---------+
        |         |           |           |         |
   [Spoke 1] [Spoke 2]   [Spoke 3]   [Spoke 4] [Spoke 5]
        \_________\_________/___________/_________/
                  (siblings link to each other)

What each role does.

• The pillar (hub). A long, comprehensive page that defines the topic, covers every subtopic at summary depth, and links out to each spoke. Typical length: 3,000-5,000 words.
• The spokes. Focused articles that go deep on one subtopic each. Each spoke links back to the pillar and sideways to 2-4 sibling spokes. Typical length: 1,200-2,500 words.
• Contextual links. Inline anchor text that names the entity being linked. "See our llms.txt guide" beats "click here" — the anchor is what AI crawlers extract as relationship context.

Why this beats flat blogs. A flat blog with 200 posts and no clusters looks like noise to an LLM. A hub-and-spoke site with 10 clusters of 8 articles each looks like 10 areas of depth.

Sizing the cluster. A pillar with only 1-2 spokes is too thin to signal authority. A pillar with 30+ spokes is hard to traverse and dilutes focus. The sweet spot is 5-15 spokes per pillar.

How Alice Labs uses it. The site you are reading sits at /en/insights/ai-search/site-architecture-for-ai-crawlers — a spoke under the AI Search Optimization Guide pillar, in the ai-search cluster. This guide links to llms.txt, schema.org, and citation playbooks; each of those links back here.

URL Structure Best Practices

In short

Clean URLs are machine-readable hierarchies. Adopt a flat semantic pattern like /insights/{cluster}/{slug}, keep it stable, and avoid query strings, dates, and category IDs. The path itself becomes a topical signal AI crawlers can parse without rendering the page.

URLs are the cheapest hierarchy you can publish. Every crawler — search, AI, or accessibility — parses them before fetching the page. They are also the most permanent: changing one breaks every inbound link.

The pattern that works.

https://alicelabs.ai/en/insights/ai-search/site-architecture-for-ai-crawlers
                       |        |              |
                    section  cluster         slug

Five rules for AI-friendly URLs.

• Use lowercase letters and hyphens. Underscores and uppercase letters are inconsistent across crawlers. Hyphens are the universally safe word separator.
• Keep the cluster name in the path. /insights/ai-search/{slug} tells a crawler the topic before it fetches the page. /insights/{slug} hides the topic.
• Avoid query strings for content. ?id=42 and ?category=ai are weak signals and easily de-duplicated. Use static paths for canonical content.
• Avoid dates in the URL. /2024/03/post-title locks content to a moment in time and discourages updates. Date-stamped paths look stale to LLMs.
• Match the slug to the primary keyword. The slug is the strongest single token in the URL. Make it descriptive and unambiguous.

Trailing slashes. Pick one convention (with or without) and stick to it. Mixed conventions create duplicate-content issues. 301-redirect the wrong variant to the right one.

Migration discipline. If you must change URLs, 301-redirect every old path to its new home and update every internal link. Broken redirect chains lose entity context that took years to build.

Internal Linking Density and Quality

In short

AI crawlers extract entity relationships from internal links. Aim for at least 3-5 contextual links per article, with descriptive anchor text that names the target entity. Density without quality is noise; quality without density leaves entities disconnected.

Internal links are how an LLM learns what a page is about beyond the page itself. A guide on llms.txt that links to robots.txt, sitemap.xml, and Schema.org is obviously about technical AI-search infrastructure. The same guide with no internal links is just text.

Three rules for AI-friendly internal linking.

• Density: 3-5 contextual links per article minimum. Every spoke should link to its pillar and 2-4 sibling spokes. Below 3 the entity graph is too sparse to traverse.
• Quality: descriptive anchor text. Anchor text is the relationship label LLMs extract. "See our llms.txt guide" beats "click here" by a wide margin.
• Placement: in-context, not just in a related-links block. Inline links inside paragraphs carry topical weight. Footer or sidebar lists are easy to ignore.

Anchor text patterns that work.

• Entity-named: "the BreadcrumbList schema" rather than "this article"
• Descriptive: "our 9-step ChatGPT citation playbook" rather than "read more"
• Question-based: "what is LLMO" rather than "definition" — useful for matching long-tail prompts

Avoiding the orphan trap. An orphan page is one with no inbound internal links. AI crawlers find it only via the sitemap, never via context. Run a crawl after every architecture change and confirm zero orphans.

Avoiding link cannibalization. If two articles target the same primary keyword, internal links will split authority between them. Designate one as canonical for the keyword and link the other one to it, or merge them.

llms.txt + sitemap.xml + robots.txt — the Technical Trifecta

In short

Three text files at the domain root expose your architecture to crawlers. robots.txt is the gatekeeper, sitemap.xml is the full discovery map, and llms.txt is the curated highlights. AI crawlers read all three; serious sites publish all three.

Each file does a different job. They are complementary, not substitutes.

robots.txt — the gatekeeper. Plain text at /robots.txt. Tells crawlers what they may access. Honored on a voluntary basis by major AI providers (OpenAI, Anthropic, Google, Perplexity).

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

sitemap.xml — the full discovery map. XML at /sitemap.xml. Lists every canonical URL on the site with optional lastmod and priority. Required for full coverage by both search and AI crawlers.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://yourdomain.com/en/insights/ai-search/site-architecture-for-ai-crawlers</loc>
    <lastmod>2026-05-06</lastmod>
  </url>
  <!-- one entry per canonical URL -->
</urlset>

llms.txt — the curated highlights. Markdown at /llms.txt. Proposed September 3, 2024 by Jeremy Howard at Answer.AI. Full spec at llmstxt.org. Lists the high-value subset of pages with descriptions, organized by cluster.

# Alice Labs

> Nordic AI consultancy specializing in LLMO and enterprise AI search.

## AI Search

- [AI Search Optimization Guide](/en/insights/ai-search-optimization-guide): pillar guide
- [Site Architecture for AI Crawlers](/en/insights/site-architecture-for-ai-crawlers): the topical authority pattern
- [llms.txt Guide 2026](/en/insights/llms-txt-guide-2026): how to write the file

## Schema and Structured Data

- [Schema.org for AI](/en/insights/schema-org-for-ai): structured data LLMs read

BreadcrumbList schema — the per-page hierarchy. Inside each page, JSON-LD BreadcrumbList markup expresses the cluster path. Place it once per page; it pairs with the URL structure.

{
  "@context": "https://schema.org",
  "@type": "BreadcrumbList",
  "itemListElement": [
    { "@type": "ListItem", "position": 1, "name": "Insights",
      "item": "https://yourdomain.com/en/insights" },
    { "@type": "ListItem", "position": 2, "name": "AI Search",
      "item": "https://yourdomain.com/en/insights/ai-search" },
    { "@type": "ListItem", "position": 3, "name": "Site Architecture for AI Crawlers" }
  ]
}

The reconciliation rule. Whatever you allow in robots.txt must also be reachable from sitemap.xml and (for high-value pages) listed in llms.txt. Inconsistencies between the three files are a common audit failure.