Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    answer-readiness/snippet-gate-coverage

    Snippet-gate coverage analysis

    What it checks

    Computes the site’s effective snippet permissions per crawler, by merging <meta name="robots">, per-bot meta tags and X-Robots-Tag response headers. It then measures those permissions against the page’s actual answer content. Is max-snippet numerically smaller than the primary answer span? And does data-nosnippet coverage overlap the answer span, the FAQ answers, or the main-content tables? Reports the specific suppressed text, not just the directive.

    Why it matters

    Google states the eligibility gate directly: to appear as a supporting link, a page ‘must be indexed and eligible to be shown in Google Search with a snippet’. It names nosnippet, data-nosnippet, max-snippet and noindex as the controls that limit what AI Overviews and AI Mode can show (S4). This makes the causal chain fully documented rather than inferred. A max-snippet value shorter than the answer sentence truncates the answer below usefulness. data-nosnippet wrapping the answer removes it from AI surfaces entirely, while leaving it visible to humans. That failure is invisible to page-level SEO reports, because the directive itself is technically ‘valid’.

    Evidence

    • MCP Specification 2026-07-28 — Tools — Model Context Protocol (spec, URL verified 2026-08-20)

    • tools/list result set MUST NOT vary per-connection or as a side effect of other requests (MAY vary by authorization). Servers SHOULD return tools in deterministic order — rationale given verbatim: enables client caching and ‘improves LLM prompt cache hit rates’. inputSchema MUST be a valid JSON Schema object (not null); defaults to JSON Schema 2020-12. Tool names SHOULD be 1-128 chars, case-sensitive, only [A-Za-z0-9_.-], unique within a server. Full x-mcp-header constraint list including static-reachability rule (chain of only properties keys; never through items/oneOf/anyOf/allOf/not/if/then/else/$ref). Clients MUST exclude violating tools from tools/list. If outputSchema present, servers MUST conform. Clients MUST treat annotations as untrusted.

    • Which per-bot names are read (verified 2026-10-20). A directive counts only under a name its crawler’s operator documents reading.

    • Google: “Google supports two user agent tokens in the robots meta tag; other values are ignored” — googlebot and googlebot-news. For the header, a user agent “may optionally” precede the rules — developers.google.com/…/robots-meta-tag (verified 2026-10-20). Google-Extended is “a standalone product token” for robots.txt and “doesn’t have a separate HTTP request user agent string” — developers.google.com/…/google-common-crawlers (verified 2026-10-20).

    • Bing: “you can replace the generic name="robots" with name="bingbot"”, and each tag “can also be specified as X-Robots-Tag” — bing.com/…/robots-meta-tags-and-attributes-that-bin… (verified 2026-10-20). Bing’s crawler list names no msnbot — bing.com/…/which-crawlers-does-bing-use-8c184ec0 (verified 2026-10-20).

    • Apple: <meta name="applebot" content="nosnippet"> and X-Robots-Tag: applebot: nosnippet. “Apple will not use data tagged nosnippet as additional context and up-to-date content when AI models are used to generate output” — support.apple.com/…/119829 (verified 2026-10-20).

    • OpenAI, Anthropic, Perplexity and Common Crawl document robots.txt controls only — developers.openai.com/…/bots, support.claude.com/…/8896518, docs.perplexity.ai/…/bots, commoncrawl.org/ccbot (all verified 2026-10-20).

    How it scores

    Tier per evidence policy: scored — grade A meets the A/B bar required for scored audits.

    Example finding

    A publisher sets <meta name="robots" content="max-snippet:80"> site-wide to discourage scraping, then invests in FAQPage markup. Every answer span on the site runs 150-250 characters, so the AI-surface snippet budget cuts each answer roughly at its midpoint, mid-clause. Separately, a support site wraps its answer bodies in <div data-nosnippet>, inherited from an old ‘prevent snippet theft’ template. nosnippetCoverage is 0.71, and every acceptedAnswer is inside it. The pages are therefore indexed, visibly correct to humans, and structurally invisible to AI Overviews.

    Sources