Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    machine-discovery/discovery-index-coverage

    Pages are covered by a discovery index

    What it checks

    Every scanned page must appear in at least one discovery index: the sitemap (including the sub-sitemaps a <sitemapindex> points at) or the llms.txt link list. A page in neither is reachable only through the link graph, which the non-JS-executing AI crawlers may not traverse.

    State Result
    every page found in an index pass
    >50% of pages found in no index fail, priority medium
    ≤50% of pages found in no index warn, priority low
    no sitemap URLs and no llms.txt links at all warn, priority medium
    no pages scanned na

    Both sides of the comparison go through one key — host without www., lower-cased decoded path, no trailing slash, no scheme, query or fragment — and a page also matches on its declared <link rel="canonical">.

    Why it matters

    A valid sitemap.xml referenced from robots.txt, with absolute URLs and accurate <lastmod>, increases the set of URLs that AI-feeding crawlers discover and the speed at which changed pages are re-fetched.

    Evidence

    sitemap.xml for AI crawler discovery

    Sitemaps are a stable, universally-implemented de facto standard (protocol 0.9, 50k URLs / 50MB limits, robots.txt Sitemap: discovery). Google documents consuming them directly: ‘Google reads your sitemap regularly, so be sure to include all the content that you want Google to crawl’. It also recommends <lastmod>. And Google’s AI-features guidance makes AI Overviews and AI Mode eligibility conditional on ordinary Search indexing, so sitemap-driven discovery transitively feeds an AI surface.

    The same applies through Bing’s index, which grounds Copilot. Vercel’s crawl-waste data supplies an indirect argument. ChatGPT wastes 34.82% of fetches on 404s, and Claude 34.16%, against Googlebot’s 8.22%. That is the signature of crawlers working from stale link graphs — precisely the failure a current sitemap with accurate lastmod mitigates.

    Limits

    No AI crawler vendor documents sitemap consumption. OpenAI’s bots page, Anthropic’s crawler article and Perplexity’s docs never mention sitemaps, and Google’s own AI-features page insists there are ‘no additional technical requirements’ for AI features. Server-log reports that GPTBot and ClaudeBot request /sitemap.xml exist but are single-site blog analyses, not controlled experiments — treat as suggestive only. The defensible framing is: sitemaps are proven for the Google/Bing indexes that ground several AI answer surfaces, and unproven-but-plausible for the direct AI crawlers. Note also that <lastmod> must be the page’s real modification date, not the generation date, or the signal is actively misleading.

    How it scores

    Sitemaps are a stable, universally implemented de facto standard. Google documents consuming them directly: “Google reads your sitemap regularly, so be sure to include all the content that you want Google to crawl.” That reaches AI Overviews and AI Mode through the same index. That is one documented consumer, not the AI vendors themselves: OpenAI’s bots page, Anthropic’s crawler article and Perplexity’s documentation never mention sitemaps, and Google’s own AI-features page insists there are “no additional technical requirements”. Server logs showing GPTBot and ClaudeBot fetching /sitemap.xml are single-site reports. Well-established mechanism, weak AI-specific proof, is grade B.

    Sources