Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/robots-directives

    Robots directives do not block AI indexing

    What it checks

    Every robots directive that applies to a scanned page, from both transports and from every directive-carrying meta name:

    • <meta name="robots"> and the per-bot names an operator documents reading: googlebot and googlebot-news (Google), bingbot (Bing) and applebot (Apple). Tags are read off the DOM, so all tags count rather than the last one;
    • the X-Robots-Tag response header, including the per-bot X-Robots-Tag: googlebot: noindex form, scoped to the same names.

    Blocking or snippet-suppressing directives addressed to a name no operator documents reading (gptbot, oai-searchbot, chatgpt-user, claudebot, claude-searchbot, claude-user, perplexitybot, ccbot, google-extended, applebot-extended, msnbot) are named in the message and never scored. A bot prefix in the header scopes every directive after it.

    Values are tokenized on commas and compared token-wise, so none is recognised as noindex+nofollow and noindexifembedded is not mistaken for a full block.

    State Result
    a non-utility page carries noindex/none fail — critical when the homepage is among them, otherwise high
    a non-utility page carries nosnippet, max-snippet:0, or noarchive without nocache warn, priority medium — indexed but unquotable
    a non-utility page carries nocache (alone or with noarchive) pass, naming it — Bing still shows URL, title and snippet
    only utility routes (cart, login, search, account, …) carry noindex pass, naming them
    a directive sits only under an undocumented per-bot name (e.g. gptbot) verdict unaffected; the message names the tag
    no directive blocks or suppresses pass
    no pages scanned na

    Why it matters

    A page carrying noindex in a robots meta tag, or in the equivalent X-Robots-Tag header, is excluded from Google’s Search index. Google documents that the same family of robots directives governs whether the page can be shown, or used as a direct input, in AI Overviews and AI Mode.

    Evidence

    • Google Search Central, robots meta tag reference. noindex: “Do not show this page, media, or resource in search results.” nosnippet: it “will also prevent the content from being used as a direct input for AI Overviews and AI Mode”. max-snippet:[number]: it “will also limit how much of the content may be used as a direct input for AI Overviews and AI Mode”. The doc scopes these rules to “all forms of search results (at Google: web search, Google Images, Discover, AI Overviews, AI Mode)” — developers.google.com/…/robots-meta-tag (verified 2026-08-21)
    • Google Search Central, AI features and your website: site owners may use the nosnippet, data-nosnippet, max-snippet or noindex controls to limit how their content appears in AI features on Search. Eligibility for AI Overviews and AI Mode carries “no additional technical requirements” beyond ordinary Search eligibility — that is, the AI surfaces are fed by the same Search index that noindex removes the page from — developers.google.com/…/ai-features (verified 2026-08-21)
    • The directive is a long-standing, cross-vendor indexing control rather than a Google-only convention; RFC 9309 governs the robots.txt half of the same exclusion family and is a ratified standard with named crawler consumers — rfc-editor.org/…/rfc9309.html (verified 2026-08-21)

    Per-bot meta names (verified 2026-10-20). Only a name the crawler’s operator documents reading can change a verdict.

    So <meta name="GPTBot" content="noindex"> and its kind bind no documented consumer. The audit names them in its message and does not score them. The grade stays A: it rests on the generic robots name and the header, which Google, Bing and Apple document.

    Limits

    noindex gates index-derived surfaces only. User-initiated fetchers are unaffected: OpenAI documents that for ChatGPT-User “Because these actions are initiated by a user, robots.txt rules may not apply” (developers.openai.com/…/bots), and Perplexity documents that Perplexity-User “Generally ignores robots.txt rules” (docs.perplexity.ai/…/bots) — both verified 2026-08-21. A noindexed page can therefore still be fetched and quoted live in a chat answer, so the audit’s failure language (“blocking AI crawlers”) overstates the effect for the conversational-agent path.

    The snippet controls are only partly instrumented. The audit reads nosnippet and max-snippet:0 and warns on them, because Google documents that they keep content out of AI Overviews and AI Mode (developers.google.com/…/robots-meta-tag, verified 2026-10-17). Google ignores noarchive and nocache; Bing reads both, and its two sources treat them differently. Bing’s robots meta tag reference says noarchive means “Do not link in Chat and Copilot”, and nocache means “Display only URL/Snippet/Title in Chat or Copilot”.

    It adds: “If content has both NOCACHE and NOARCHIVE tags, we will treat it as NOCACHE.” (bing.com/…/robots-meta-tags-and-attributes-that-bin…, verified 2026-10-17). The 2023 announcement says the same in other words: “Content tagged NOARCHIVE will not be included in Bing Chat answers, not be linked to in the answers”, while “Content with the NOCACHE tag may be included in Bing Chat answers. We will only display URL/Snippet/Title in the answer” (blogs.bing.com/…/Announcing-new-options-for-webmast…, verified 2026-10-17).

    So noarchive alone keeps a page out of Copilot answers, and the audit warns on it. nocache limits what Copilot shows but still links and quotes the page’s snippet, so it does not forbid quoting. The audit names a nocache page in its pass message and does not warn. A page carrying both is treated as nocache, as Bing does. Until 2026-10-17 the audit warned on nocache as if it forbade quoting; no source supported that.

    It does not read element-level data-nosnippet, and it does not judge a positive max-snippet limit, which narrows rather than removes what an AI answer may quote.

    How it scores

    The consumer is named in vendor documentation. Google Search Central states that noindex removes the page from search results. It states that nosnippet and max-snippet “will also prevent” or “limit” the content being “used as a direct input for AI Overviews and AI Mode”.

    Sources