Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/robots-directives

    Robots directives do not block AI indexing

    What it checks

    Every robots directive that applies to a scanned page, from both transports and from every directive-carrying meta name:

    • <meta name="robots"> and the per-bot forms (googlebot, google-extended, gptbot, oai-searchbot, claudebot, applebot-extended, bingbot, perplexitybot, …), read off the DOM so all tags count rather than the last one;
    • the X-Robots-Tag response header, including the per-bot X-Robots-Tag: googlebot: noindex form.

    Values are tokenized on commas and compared token-wise, so none is recognised as noindex+nofollow and noindexifembedded is not mistaken for a full block.

    State Result
    a content page carries noindex/none fail — critical when the homepage is among them, otherwise high
    a content page carries nosnippet, noarchive or max-snippet:0 warn, priority medium — indexed but unquotable
    only utility routes (cart, login, search, account, …) carry noindex pass, naming them
    no directive blocks or suppresses pass
    no pages scanned na

    Why it matters

    A page carrying noindex in a robots meta tag, or in the equivalent X-Robots-Tag header, is excluded from Google’s Search index. Google documents that the same family of robots directives governs whether the page can be shown, or used as a direct input, in AI Overviews and AI Mode.

    Evidence

    • Google Search Central, robots meta tag reference. noindex: “Do not show this page, media, or resource in search results.” nosnippet: it “will also prevent the content from being used as a direct input for AI Overviews and AI Mode”. max-snippet:[number]: it “will also limit how much of the content may be used as a direct input for AI Overviews and AI Mode”. The doc scopes these rules to “all forms of search results (at Google: web search, Google Images, Discover, AI Overviews, AI Mode)” — developers.google.com/…/robots-meta-tag (verified 2026-08-21)
    • Google Search Central, AI features and your website: site owners may use the nosnippet, data-nosnippet, max-snippet or noindex controls to limit how their content appears in AI features on Search. Eligibility for AI Overviews and AI Mode carries “no additional technical requirements” beyond ordinary Search eligibility — that is, the AI surfaces are fed by the same Search index that noindex removes the page from — developers.google.com/…/ai-features (verified 2026-08-21)
    • The directive is a long-standing, cross-vendor indexing control rather than a Google-only convention; RFC 9309 governs the robots.txt half of the same exclusion family and is a ratified standard with named crawler consumers — rfc-editor.org/…/rfc9309.html (verified 2026-08-21)

    Limits

    noindex gates index-derived surfaces only. User-initiated fetchers are unaffected: OpenAI documents that for ChatGPT-User “Because these actions are initiated by a user, robots.txt rules may not apply” (developers.openai.com/…/bots), and Perplexity documents that Perplexity-User “Generally ignores robots.txt rules” (docs.perplexity.ai/…/bots) — both verified 2026-08-21. A noindexed page can therefore still be fetched and quoted live in a chat answer, so the audit’s failure language (“blocking AI crawlers”) overstates the effect for the conversational-agent path. Separately, the strongest AI-specific directives Google names — nosnippet, data-nosnippet, max-snippet:0 — are outside what the current implementation reads, so the A-grade mechanism is only partially instrumented.

    How it scores

    The consumer is named in vendor documentation. Google Search Central states that noindex removes the page from search results. It states that nosnippet and max-snippet “will also prevent” or “limit” the content being “used as a direct input for AI Overviews and AI Mode”.

    Sources