Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    machine-discovery/sitemap-absolute-urls

    Sitemap uses absolute URLs

    What it checks

    Sitemap URLs must be absolute (starting with https://) so AI crawlers can resolve them without ambiguity.

    Why it matters

    A valid sitemap.xml referenced from robots.txt, with absolute URLs and accurate <lastmod>, increases the set of URLs that AI-feeding crawlers discover and the speed at which changed pages are re-fetched.

    Evidence

    sitemap.xml for AI crawler discovery

    Sitemaps are a stable, universally-implemented de facto standard (protocol 0.9, 50k URLs / 50MB limits, robots.txt Sitemap: discovery). Google documents consuming them directly: ‘Google reads your sitemap regularly, so be sure to include all the content that you want Google to crawl’. It also recommends <lastmod>. And Google’s AI-features guidance makes AI Overviews and AI Mode eligibility conditional on ordinary Search indexing, so sitemap-driven discovery transitively feeds an AI surface.

    The same applies through Bing’s index, which grounds Copilot. Vercel’s crawl-waste data supplies an indirect argument. ChatGPT wastes 34.82% of fetches on 404s, and Claude 34.16%, against Googlebot’s 8.22%. That is the signature of crawlers working from stale link graphs — precisely the failure a current sitemap with accurate lastmod mitigates.

    Limits

    No AI crawler vendor documents sitemap consumption. OpenAI’s bots page, Anthropic’s crawler article and Perplexity’s docs never mention sitemaps, and Google’s own AI-features page insists there are ‘no additional technical requirements’ for AI features. Server-log reports that GPTBot and ClaudeBot request /sitemap.xml exist but are single-site blog analyses, not controlled experiments — treat as suggestive only. The defensible framing is: sitemaps are proven for the Google/Bing indexes that ground several AI answer surfaces, and unproven-but-plausible for the direct AI crawlers. Note also that <lastmod> must be the page’s real modification date, not the generation date, or the signal is actively misleading.

    How it scores

    Sitemaps are a stable, universally implemented de facto standard. Google documents consuming them directly: “Google reads your sitemap regularly, so be sure to include all the content that you want Google to crawl.” That reaches AI Overviews and AI Mode through the same index. That is one documented consumer, not the AI vendors themselves: OpenAI’s bots page, Anthropic’s crawler article and Perplexity’s documentation never mention sitemaps, and Google’s own AI-features page insists there are “no additional technical requirements”. Server logs showing GPTBot and ClaudeBot fetching /sitemap.xml are single-site reports. Well-established mechanism, weak AI-specific proof, is grade B.

    Sources