Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/bot-content-delta-declared

    Content served to AI crawler User-Agents matches the browser, or is declared

    What it checks

    Measures whether the site serves materially different content to AI crawler user-agents than to a browser, and — when it does — whether that difference is declared with the structured data Google specifies for restricted content. Undeclared UA-conditional serving is cloaking, and it also means answer engines cite your paywall stub instead of your article.

    When a crawler request gets less, the audit also fetches the URL once as an unrecognised control bot. If the control bot gets the same reduced page, the reduction is bot management aimed at every unidentified client, not a page keyed on the crawler’s name. That warns rather than fails.

    Why it matters

    Google states that isAccessibleForFree: false with hasPart/cssSelector markup ‘helps Google differentiate paywalled content from the practice of cloaking, which violates spam policies’ (s15) — that is, serving a crawler less than a user is sanctioned only when declared. Falsifiable and directly measurable: extract main text for URL U under a browser UA and under crawler UA C; if len(text_C)/len(text_browser) falls below threshold (or shingle Jaccard drops below ~0.7), the site conditions content on UA.

    The declaration is equally checkable — and, importantly, the declared cssSelector must match a real element in the served HTML, which is where most implementations silently fail. The condition is on the User-Agent naming the crawler. A reduction an unrecognised bot gets as well is not conditioned on that name, so it is not the delta Google’s rule addresses.

    Evidence

    • Paywalled content structured data — Google (vendor-doc, URL verified 2026-08-20)
    • isAccessibleForFree: false plus hasPart with @type: WebPageElement, isAccessibleForFree: false and a cssSelector naming the restricted region. Applies to CreativeWork subtypes (Article, NewsArticle, Blog, Course, HowTo, Review, WebPage, Comment, Message). Google states this markup “helps Google differentiate paywalled content from the practice of cloaking, which violates spam policies” — i.e. serving less content to a crawler is only safe when declared.
    • OpenAI crawlers and user agents — OpenAI (vendor-doc, URL verified 2026-08-20)
    • Exact UA strings and published IP-range JSONs (re-verified 2026-10-19). OAI-SearchBot (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) … Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot) → openai.com/searchbot.json. GPTBot (…compatible; GPTBot/1.4; +https://openai.com/gptbot) → openai.com/gptbot.json. ChatGPT-User (…compatible; ChatGPT-User/1.0; +https://openai.com/bot) → openai.com/chatgpt-user.json. OAI-AdsBot → openai.com/adsbot.json. All four JSON endpoints return HTTP 200 (curl-verified). No mention of Web Bot Auth. Note Google-Extended has no UA at all, so it cannot be probed by request.

    Limits

    • Every probe leaves the scanner’s own address with an AI crawler’s name in the User-Agent. It shows how the site answers a request that names the AI crawler. It does not show what the real crawler receives.
    • Real bots are told apart by network or signature, not by the string. Cloudflare grants verified status to a bot that “declares who it is deterministically, through a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS” (Cloudflare verified bots policy (verified 2026-10-19)). OpenAI publishes one IP-range file per bot, for example openai.com/gptbot.json and openai.com/chatgpt-user.json (OpenAI crawlers and user agents (verified 2026-10-19)). Google, whose crawlers this audit does not probe, illustrates the same rule: it documents a reverse-DNS check, or a match against its published IP ranges (Verifying Google crawlers and fetchers (verified 2026-10-19)). The scanner can present none of these.
    • The delta this audit reports is UA-keyed. A site that serves the real crawler by verified identity can give it a different page than the probe gets.
    • A thinner 200 can be bot management aimed at unidentified clients. For GPTBot, a training crawler, Cloudflare’s legacy Block AI bots setting, labelled “Deprecating on September 15, 2026”, covers “additional unverified bots” as well as verified training bots (Cloudflare Block AI bots (verified 2026-10-19)). Cloudflare’s own example rules do exactly this: (cf.bot_management.verified_bot) skips all remaining custom rules, so “Known good bots … bypass all custom rules”, and (cf.bot_management.score lt 30 and not cf.bot_management.verified_bot) blocks other automated traffic (Cloudflare Challenge bad bots (verified 2026-10-20)).

    Super Bot Fight Mode likewise gives “Definitely automated traffic”, “Likely automated traffic” and “Verified bots” separate responses (Cloudflare Super Bot Fight Mode (verified 2026-10-20)), so the verified crawler can get the full page while unidentified bots get less.

    • The control User-Agent separates the two. If the unrecognised control bot gets the same reduced page as the crawler request, the delta warns. “The same” is checked against the crawler’s page: the control text falls short of the browser text, is 80% to 125% of the crawler text’s length, and shares its shingles at 0.7 or more. A control that gets some other reduction, such as a challenge or an empty shell, shows nothing, and the delta still fails.

    If the control bot gets the full page, the reduction is keyed on the crawler’s name and still fails. A rule keyed on the name and on verified identity together, serving a stub to look-alikes only, still reads as a name-keyed delta; see Deferred.

    • A thin 200 that carries a bot-challenge marker is not compared. It is a refusal, which access-crawl-control/ai-crawler-edge-parity reports.

    How it scores

    Tier per evidence policy: scored — grade A meets the A/B bar required for scored audits.

    An undeclared delta keyed on the crawler’s name, a declared selector that matches nothing, or a bot-only longer page fails at High. An undeclared delta whose reduced page the unrecognised control bot gets as well (80% to 125% of the crawler text’s length, shingle similarity at least 0.70) warns at Medium. A declared delta whose selector resolves passes.

    Example finding

    An article renders 6,200 chars of body text to Chrome. The same URL fetched as GPTBot/1.4 returns 200 with 480 chars — headline, dek, and a subscribe prompt — and the page’s JSON-LD NewsArticle omits isAccessibleForFree entirely. The site reads as cloaking to Google’s spam policy, and ChatGPT/Perplexity ingest and cite the 480-char stub as if it were the article.

    Sources