AI Crawlability Consulting: Get Discovered by ChatGPT & Copilot

What AI Crawlability Consulting Actually Fixes in 2026

In short

AI crawlability consulting fixes three layers most enterprise sites break silently: bot access (robots.txt, WAF, llms.txt), retrieval index (Bing coverage, sitemap health), and entity resolution (schema.org sameAs, Wikidata, per-item stable @ids). The problem is not that sites are down — they are technically live, and traffic dashboards look fine — but that AI answer engines cannot access, index, or disambiguate them.

Traditional search is contracting. Gartner projects a 25% drop in traditional search engine volume by 2026 as users shift to AI chatbots and virtual agents. ChatGPT Search processes an estimated 250 to 500 million weekly queries; Microsoft Copilot grounds every answer in Bing's live index; Perplexity, Claude, and Google AI Overviews each occupy their own retrieval surfaces. None of them use the same signals that Googlebot has taught the industry to optimise for.

The Data-Mania 2026 B2B SaaS benchmark found that 90% of B2B SaaS brands have zero AI search mentions. That is not a ranking problem in the classical sense — those brands are not competing for position 4 versus position 5. They are invisible. The pages the LLM would cite either cannot be fetched, are not in the retrieval index the LLM uses, or cannot be resolved to a stable entity the LLM will trust.

Alice Labs frames the discipline as three layers, in the order that a page fails:

• Bot access — can OpenAI's OAI-SearchBot, Anthropic's ClaudeBot, PerplexityBot, and Bingbot actually fetch the page? robots.txt is the visible layer, but WAF and Cloudflare bot management block many crawlers by user-agent heuristic even when robots.txt allows them.
• Retrieval index — is the page in Bing? In Google's index? Is it in the sitemap Google and Bing actually crawl? IndexNow submission and Bing Webmaster Tools verification are the highest-leverage single fixes.
• Entity resolution — when the LLM decides to cite something on this topic, can it identify your organisation as the source and not confuse you with a same-named entity? This is what schema.org and the sameAs chain are for.

A consulting engagement typically starts with an audit against the 12 structural citation features Alice Labs uses on its own site (we enumerate them in section 6 of this guide). The point is not to add every feature at once but to identify which of the three layers is failing first — because the ROI ordering is strict: bot access before index, index before entity, entity before content.

The Three OpenAI Crawlers You Have to Configure Correctly

In short

OpenAI runs three distinct crawlers with independent purposes: GPTBot handles model training; OAI-SearchBot fetches pages shown as ChatGPT Search citations; ChatGPT-User fetches on demand when a user asks ChatGPT to visit a URL. They honor independent robots.txt rules, so you can block GPTBot (training opt-out) while allowing OAI-SearchBot (visibility). Robots.txt changes propagate through OpenAI's crawler caches in roughly 24 hours.

The most expensive misconfiguration Alice Labs sees in audits is not blocking too much — it is blocking the wrong crawler. Teams that "opted out of OpenAI" by adding User-agent: GPTBot / Disallow: / to robots.txt often also blocked OAI-SearchBot, which is the crawler that fetches pages ChatGPT actually cites. The result: they preserved a training preference they may not have needed and simultaneously turned off the visibility surface that mattered.

The three crawlers are distinct products:

• GPTBot — OpenAI's model-training crawler. Block this to opt out of training data collection. Blocking GPTBot has no effect on ChatGPT Search visibility.
• OAI-SearchBot — the crawler that fetches pages surfaced as citations inside ChatGPT Search. If AI visibility is the goal, this is the one you cannot accidentally block.
• ChatGPT-User — an on-demand fetcher, invoked when a user explicitly asks ChatGPT to visit a URL (browse tool). Allowing it is polite; blocking it breaks the "visit this link" user experience.

OpenAI publishes JSON IP-range files for each crawler at openai.com/gptbot.json, openai.com/searchbot.json, and openai.com/chatgpt-user.json. WAFs and CDN bot rules should allowlist those ranges alongside the user-agent match; Alice Labs regularly audits engagements where Cloudflare or an enterprise WAF was silently rejecting OAI-SearchBot despite a permissive robots.txt.

One operational note that saves debugging time: robots.txt changes for OpenAI crawlers take approximately 24 hours to propagate through OpenAI's caches. Do not judge the effect of a change on the same day you deploy it.

# Block training, allow ChatGPT Search visibility
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

Do not paste that block without checking your existing rules — cascading User-agent: * directives will still apply, and the ordering matters. This is one of the checks in a proper crawlability audit.

Bing Indexing Is the Binary Gate for ChatGPT and Copilot

In short

Microsoft Copilot grounds every answer in Bing's live search index, and ChatGPT Search uses Bing as a major retrieval source. Pages not in Bing's index are effectively invisible to both regardless of Google ranking — it is binary, not a spectrum. IndexNow (submit URLs at publish time) accelerates Bing indexation from days to minutes, with a quota of 10,000 URL submissions per day per verified property.

Google ranking and LLM citation are different scoreboards. A page can rank in the Google top 3 for its target keyword and still get zero ChatGPT Search or Copilot citations — because ChatGPT Search does not use Google's index, and Copilot definitely does not. Both retrieval systems rely heavily on Bing.

The consequence is uncomfortable for teams whose measurement stops at Google: Bing absence is a hard zero for Copilot delivery, and Copilot represents roughly 40% of enterprise LLM usage in Microsoft-first shops. If your page is not in Bing, you are not in Copilot answers, regardless of how much Google authority you have accumulated.

Two levers matter, in order:

1. Bing Webmaster Tools verification — the prerequisite. Without it, you cannot see coverage data, cannot submit URLs at scale, and cannot use IndexNow. This is the first infrastructure step of every Alice Labs consulting engagement, before schema work.
2. IndexNow submission — Microsoft's open protocol for pinging Bing (and Yandex, Naver, Seznam) at publish time. Bing accepts up to 10,000 URL submissions per day per verified property. In Alice Labs' measured engagements, IndexNow shifts new-content indexation from days to minutes.

Alice Labs' internal proof is concrete. When Bing coverage rose from partial to full across our own 300+ articles — the result of Bing Webmaster verification, sitemap generation, and IndexNow rollout — LLM citation surface expanded within 14 days. The mechanism is not mysterious: LLMs cite what is in the index; the index gets more of your pages; more pages get cited.

A common failure mode worth naming: Bing Webmaster's URL coverage report can show a page as "discovered" without it being retrievable by Copilot. "Discovered but not crawled" and "Crawled but not indexed" are separate diagnostic categories, and each has a distinct fix path. A crawlability audit that stops at "submitted to IndexNow" is incomplete.

robots.txt, llms.txt, and the AI Crawler Access Layer

In short

The minimum viable AI crawler allowlist is GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, and Bingbot. llms.txt adoption reached ~14% of the top 100K sites by June 2026 (Cloudflare Radar) and Princeton's GEO-bench found ~23% more citations for sites with well-structured llms.txt versus matched controls — but no major LLM provider has publicly committed to consuming it in production retrieval. Treat it as a low-cost hedge, not a substitute for HTML crawlability.

The AI crawler access layer is two artifacts and one operational risk. The two artifacts are robots.txt — the mature, universally understood control — and llms.txt, an emerging convention that is still finding its footing. The operational risk is your WAF quietly overriding both.

The minimum viable AI allowlist in 2026 covers eight named user agents:

• OpenAI: GPTBot (training), OAI-SearchBot (Search citations), ChatGPT-User (on-demand)
• Anthropic: ClaudeBot
• Perplexity: PerplexityBot
• Google: Google-Extended (Gemini training opt-in), Applebot-Extended (Apple Intelligence)
• Microsoft: Bingbot (indexing prerequisite for Copilot and ChatGPT Search)

llms.txt is the more speculative surface. Cloudflare Radar's June 2026 report puts adoption at ~14% of the top 100K sites, up from about 2% a year earlier. The W3C opened a working draft on 16 June 2026. Princeton's GEO-bench (May 2026) measured a ~23% citation lift for sites with well-structured llms.txt against matched controls.

The awkward truth: no major LLM provider — not OpenAI, not Anthropic, not Google, not Microsoft — has publicly committed to consuming llms.txt in production retrieval. The Princeton lift may be a correlation with other quality signals rather than a causal effect of llms.txt itself. Alice Labs' policy is to publish llms.txt on every client engagement because the cost is roughly nil, while insisting that HTML crawlability, Bing indexation, and entity graph do the actual work.

The operational risk that shows up in more than half of Alice Labs' initial audits: WAF or Cloudflare bot management silently blocks OAI-SearchBot, ClaudeBot, or PerplexityBot by user-agent heuristic even when robots.txt allows them. Fixing robots.txt without checking the WAF is one of the reasons "we allow all AI crawlers already" engagements still turn up zero citations. Verify the fetch actually completes with a live request from the crawler's published IP range.

Entity Graph and sameAs: Why Identity Is a Crawlability Problem

In short

Entity resolution is the third crawlability layer. Schema.org sameAs anchors your Organization node to Wikidata, LinkedIn, Crunchbase, G2, Trustpilot, Clutch, and Google Business Profile so retrieval systems can resolve one entity across sources. A consistent entity graph delivers roughly a 20% uplift in AI Overview visibility (OrganiKPI 2026). Every listicle item needs a stable @id — the same vendor across every article resolves to the same entity.

Crawlability is not just about fetching bytes. If an LLM cannot decide who you are — distinguish Alice Labs (Stockholm enterprise AI consultancy) from a same-named company in a different country — it will not cite you, because a wrong attribution is worse than a missing one. Entity resolution is a crawlability problem in the LLM-serving sense of the word.

The primary tool is schema.org sameAs. On your Organization node, list the external authoritative profiles that unambiguously identify you:

• Wikidata — the identity anchor most LLMs treat as authoritative
• LinkedIn Company page
• Crunchbase profile
• Trustpilot, G2, Clutch — externally attributed review surfaces
• Google Business Profile
• Wikipedia article (when it exists)

Alice Labs runs its own entity graph as the worked example we replicate for clients: Wikidata IDs for our leadership — Eric Lundberg (Q140369978), Linus Ingemarsson (Q140369914), and CEO Alice Holmgren. The purpose is not vanity. It is that when a GPT-5.5 answer resolves "Alice Labs Nordic AI consultancy" against its embeddings, the Wikidata → LinkedIn → Crunchbase chain gives it a single confident entity to cite.

Every listicle and comparison article needs a related pattern: per-item stable @ids. Alice Labs' internal helper is entityIdFor(slug), which returns the pattern https://alicelabs.ai/entity/<slug>. Same vendor across every article gets the same @id. This is what lets an LLM connect "vendor X in our listicle 12 months ago" to "vendor X in this new comparison" and treat them as one entity. Without stable per-item @ids, every article is a fresh universe and no cross-article authority accumulates.

The negative rule matters just as much: never emit self-asserted aggregateRating in your JSON-LD. Google's self-serving reviews policy (originally 2019, tightened in March 2026) treats it as a manual-action risk. Alice Labs only emits ratings when they are sourced from an externally attributed third-party widget — Trustpilot, G2, or Clutch — with Review.author set to that widget's organisation. Consultants who add aggregateRating to your schema without a widget source are exposing you to a Google penalty.

The 12 Structural Citation Features Alice Labs Audits

In short

Alice Labs audits every article — its own and every client's — against a 12-feature structural rubric: answer-shaped paragraph in the first 60 words, one canonical statistic per major claim with a named primary source, named-entity density in H2s, author byline with sameAs to LinkedIn, ISO 8601 publish and last-reviewed dates in visible HTML and schema, schema.org/Table on listicles, FAQPage schema on real Q&A, Organization schema with full sameAs chain including Wikidata, Article schema referencing Organization by @id, per-item stable @id for every ItemList entry, GSC-derived keyword anchors on internal links, and external citations to authority domains.

The 12-feature rubric is the practical translation of "make it citable" into a checklist. Alice Labs runs it against every generated article via seo_knowledge.check_llm_citation_features() as a hard quality gate: 12/12 before publish, no exceptions.

The 12 structural citation features Alice Labs audits

No individual feature makes a page citable. The combined effect is a page that reads to an LLM as a well-attributed authoritative source with resolvable entities and extractable structure — which is exactly what a retrieval-and-cite system is optimised to prefer.

The most commonly missing features in Alice Labs client audits are #8 (Organization schema is present but sameAs is incomplete or missing Wikidata), #10 (listicles use fresh @ids each article, so no cross-article entity accumulates), and #11 (internal links still use "read more" and "learn more" anchors). Fixing those three alone typically moves a client from "no LLM citations" to "sporadic LLM citations" within 30 days.

MCP and the Next Crawlability Frontier: Agent-Readable Endpoints

In short

Model Context Protocol (Anthropic, open-sourced late 2024) is the emerging standard for exposing tools and data to LLM agents at query time. A public read-only MCP endpoint makes your product's data crawlable on demand — an agent can call your MCP endpoint to fetch fresh pricing, documentation, or inventory. Alice Labs advises B2B clients to publish an MCP endpoint before end of 2026 to capture agentic-search traffic, alongside — not instead of — HTML crawlability.

Passive crawlability — being fetched on a bot's schedule — is one surface. The next surface is active crawlability: agents calling your systems at query time to get fresh data. The protocol that makes this feasible without bespoke API work for every LLM vendor is Model Context Protocol (MCP), which Anthropic open-sourced in late 2024 and which crossed 200+ server implementations in 2026.

The pattern that matters for crawlability consulting is the public read-only MCP endpoint: a lightweight server that exposes structured tools like getPricing(), listServices(), or searchDocs(query). An LLM agent — Claude with the corresponding MCP client, Cursor, a Copilot workflow — calls the endpoint mid-conversation and gets a live answer instead of a stale cached web page.

Why this is a crawlability issue, not an integration issue:

• An MCP-connected agent will trust and cite the MCP tool result over an HTML page scraped a week ago. Being on the agent's tool list is the new "being in the index".
• MCP endpoints expose canonical data — a single price, a single service description, a single doc version — which eliminates the disambiguation problem that plagues HTML retrieval.
• Enterprise buyers using agents will discover you through the MCP surface first, before ever loading a marketing page.

Public MCP endpoints require clear auth boundaries and rate limits — read-only, no side effects, per-IP throttling, no PII in responses. Alice Labs applies the same operational hardening we use for our own crawlability infrastructure: monitored latency, structured logging, and a documented contract at the MCP server's manifest URL.

This is not a substitute for HTML crawlability. It is a parallel surface. Sites that skip HTML crawlability in the belief that MCP will replace it will be invisible to the ~85% of AI search traffic that still uses HTML retrieval in 2026. Ship both.

How Alice Labs Runs an AI Crawlability Consulting Engagement

In short

Alice Labs' four-week AI crawlability engagement runs in four fixed phases: Week 1 crawlability audit (robots.txt, WAF, Bing coverage, deployed vs intended schema, sameAs completeness); Week 2 entity graph rebuild (Wikidata submissions, LinkedIn/Crunchbase alignment, review-widget setup); Week 3 schema and content retrofit against the 12-feature rubric with per-item stable @ids; Week 4 IndexNow rollout, Bing Webmaster verification, top-500 URL submission, llms.txt publication. An optional 90-day measurement retainer runs weekly citation-gap tracking and monthly re-audits.

The four-week engagement is not the answer to every problem — some clients need only a two-week audit, some need a three-month retrofit — but it is the shape that produces measurable citation lift most reliably. Each week terminates in a shipped artifact and a measurable diagnostic, not a slide deck.

Alice Labs AI crawlability engagement — 4-week structure + 90-day retainer

The tools column matters. Alice Labs runs its own pipeline on the same instrumentation — llm_citation_gap.py, schema_audit.py, backlinks_analyzer.py, bing_webmaster.py — that we deploy for clients. The pipeline is battle-tested on our own 300+ articles and 100+ production AI implementations. Clients are not the R&D surface; they get the hardened version.

The single most common mid-engagement finding: a client's WAF has been silently blocking OAI-SearchBot for months, and the entire visibility problem clears once the WAF rule is fixed. The rest of the engagement is still worth doing — entity graph and schema retrofit compound over months — but the WAF fix produces measurable citation lift in the first week.

What Success Looks Like: The Alice Labs Nordic Case Study

In short

Alice Labs measured its own GPT-5.5 citation share via the llm_query_harvester (2,346 queries, 158 prompts, $64.15 cost, August 2026 baseline). 100% of Alice Labs' GPT citations in that harvest were Nordic-tagged queries — geographic specialisation compounds in LLMs. Luenendonk (DACH management consulting) validates the same pattern in a different geography, confirming geo authority as a citation moat. Consulting clients get the same measurement instrumentation as part of the 90-day retainer.

"AI crawlability consulting works" is a claim easier to make than to prove. The Alice Labs baseline is a measured artifact: the August 2026 harvester run covered 158 prompts across topical clusters we compete in, executed 2,346 GPT-5.5 queries with the Responses API web_search tool, and cost $64.15 in OpenAI credits. The output is a per-prompt log of which sources GPT-5.5 cited when asked, from which we compute per-cluster, per-intent, per-geography citation share.

The uncomfortable finding for our own team: 100% of Alice Labs' 8 GPT citations in that harvest were Nordic-tagged queries — Sweden, Nordics, Stockholm, Nordic financial services. Zero citations on global-intent queries even where Alice Labs ranks in the Google top 3. Google authority did not transfer to LLM citation on non-Nordic queries.

That is the citation-share pattern that most enterprise sites need to understand before setting an ambition. Luenendonk, the German management consulting research firm, shows the same pattern in the DACH region — dominant LLM citation share on German-language, DACH-scoped queries, thin share globally. Geographic specialisation compounds in LLMs in a way that generic global authority does not.

The Alice Labs internal target from this baseline: expand from Nordic-only citation share (8/158) to ≥30% Nordic citation share before pursuing global expansion. The reason is strategic — until we saturate the Nordic geographic moat, general-intent global content dilutes the entity signal that LLMs use to disambiguate us. The same logic applies to client engagements: identify the tightest defensible geography and vertical, saturate citations there, then expand.

Consulting clients get the same measurement instrumentation. Not opinions about citation share — actual per-prompt citation counts from an executable harvester they can rerun themselves. The 90-day retainer produces a weekly dashboard, not a monthly slide.

AI Crawlability vs Traditional Technical SEO: What Actually Overlaps

In short

Overlap: XML sitemaps, canonical tags, HTTP status hygiene, JavaScript rendering, page speed — all still matter. New for AI crawlability: entity graph completeness, per-item @id stability, MCP endpoint exposure, Bing IndexNow, llms.txt. Traditional SEO tools (Ahrefs, Semrush) infer AI Overview presence but do not measure ChatGPT/Perplexity/Copilot citations directly; most traditional agencies lack Wikidata and entity-graph practice. Alice Labs runs both surfaces from a single instrumentation stack, which is why lift attributes to specific fixes.

The confusion between technical SEO and AI crawlability is understandable — the two disciplines share half their surface area. But treating them as identical produces engagements that ship traditional SEO deliverables and wonder why LLM citations do not move.

The genuine overlap is the plumbing:

• XML sitemaps — LLMs' retrieval systems and Bing both need them
• Canonical tags — duplicate content is as toxic to LLMs as to Google
• HTTP status hygiene — no soft-404s, no chains of 301s, no 5xx flakes
• JavaScript rendering — LLM crawlers are less forgiving than Googlebot; server-side rendering is safer
• Page speed — impacts crawl budget on both surfaces

The genuine deltas are where traditional SEO agencies do not have practice:

• Entity graph completeness — Wikidata submission, sameAs chain engineering, cross-article stable @ids
• Bing indexation at scale — Bing Webmaster verification and IndexNow rollout for 10,000+ URL properties
• MCP endpoint exposure — the agentic-crawlability surface
• llms.txt — the hedge that traditional SEO tools do not audit
• Per-prompt citation-share measurement — measuring the outcome, not inferring it

The measurement gap is the biggest single reason to be sceptical of a "we do AI SEO" claim from a traditional agency. Ahrefs and Semrush infer AI Overview presence from SERP scrapes; they do not run per-prompt harvests against ChatGPT, Perplexity, or Copilot. If your consultant cannot produce a table with rows per prompt and columns per LLM showing citation counts, they are measuring the wrong thing.

Alice Labs' pitch on this axis is straightforward: we run both surfaces from a single instrumentation stack. When citation share moves, we can attribute the lift to a specific fix — a robots.txt change, a Bing coverage jump, a Wikidata entity going live — because the measurement pipeline runs continuously. Attribution is what makes the second engagement's budget approvable.

Pricing, Scope, and What to Ask a Consultant Before Signing

In short

Alice Labs offers a fixed-scope 4-week AI crawlability audit and retrofit engagement, plus an optional 90-day measurement retainer. Questions to ask any AI crawlability consultant: do you measure per-prompt citation rates (not just AI Overview appearance)? Do you have a documented entity-graph methodology including Wikidata? Do you verify Bing indexation and use IndexNow at scale? Do you build measurement dashboards or hand off PDF reports? Watch for anti-patterns: llms.txt sold as silver bullet, unsourced aggregateRating in schema, no citation-share data from the consultant's own site.

Buying AI crawlability consulting well in 2026 is a matter of asking questions the weaker vendors cannot answer. The correct posture is procurement-tight scepticism — not because the discipline is a scam, but because the influx of rebadged SEO agencies has produced a lot of misaligned promises.

The four questions Alice Labs recommends every buyer ask, in order:

1. Do you measure per-prompt citation rates? Not AI Overview appearance, not "share of voice" — actual citation counts against ChatGPT, Perplexity, and Copilot from a scripted harvester. If the answer is "we monitor AI Overviews", they are measuring Google's LLM surface only.
2. Do you have a documented entity-graph methodology including Wikidata? Ask for a case study where they submitted an entity to Wikidata and it got approved. Wikidata's approval process is opinionated; consultants who have not run the loop cannot help you clear it.
3. Do you verify Bing indexation and use IndexNow at scale? Bing Webmaster verification and 10,000-URL IndexNow submissions are the highest-ROI single lever. If the consultant treats Bing as a footnote, walk.
4. Do you build measurement dashboards you hand off? Or do you email PDF reports monthly? The former is infrastructure you own after the engagement ends; the latter creates permanent dependency and hides the underlying data.

The anti-patterns are easier to spot than the good practice. Watch for:

• Consultants who lead with llms.txt as the primary lever — the data does not support it as a substitute for HTML crawlability.
• Consultants who add aggregateRating to your schema without an external widget — this is a Google manual-action risk.
• Consultants who cannot show measured citation-share results from their own site — if they cannot cite their own citations, they cannot cite yours.

Alice Labs offers a fixed-scope four-week audit + retrofit engagement and an optional 90-day measurement retainer. Pricing is transparent — EU AI Act-native governance principles apply to our commercial terms as much as to our AI systems — and we do not charge retainers without a measured baseline first, because a retainer without a baseline is a lottery ticket for the vendor and a cost for the client.

The methodology is backed by 100+ production AI implementations shipped since 2023, and every deliverable maps to a tool in our public workflow inventory — no proprietary black box, no bespoke framework locked to us. If you want to see the full LLMO service scope, our AI search optimization guide covers the pillar this consulting service operationalises, and the best LLMO tools 2026 guide names the instrumentation stack.