Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/bot-content-delta-declared

    Content served to AI crawlers matches the browser, or is declared

    What it checks

    Measures whether the site serves materially different content to AI crawler user-agents than to a browser, and — when it does — whether that difference is declared with the structured data Google specifies for restricted content. Undeclared UA-conditional serving is cloaking, and it also means answer engines cite your paywall stub instead of your article.

    Why it matters

    Google states that isAccessibleForFree: false with hasPart/cssSelector markup ‘helps Google differentiate paywalled content from the practice of cloaking, which violates spam policies’ (s15) — that is, serving a crawler less than a user is sanctioned only when declared. Falsifiable and directly measurable: extract main text for URL U under a browser UA and under crawler UA C; if len(text_C)/len(text_browser) falls below threshold (or shingle Jaccard drops below ~0.7), the site conditions content on UA. The declaration is equally checkable — and, importantly, the declared cssSelector must match a real element in the served HTML, which is where most implementations silently fail.

    Evidence

    • Paywalled content structured data — Google (vendor-doc, URL verified 2026-08-20)
    • isAccessibleForFree: false plus hasPart with @type: WebPageElement, isAccessibleForFree: false and a cssSelector naming the restricted region. Applies to CreativeWork subtypes (Article, NewsArticle, Blog, Course, HowTo, Review, WebPage, Comment, Message). Google states this markup “helps Google differentiate paywalled content from the practice of cloaking, which violates spam policies” — i.e. serving less content to a crawler is only safe when declared.
    • OpenAI crawlers and user agents — OpenAI (vendor-doc, URL verified 2026-08-20)
    • Exact UA strings and published IP-range JSONs. OAI-SearchBot (…compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot) → openai.com/searchbot.json. GPTBot (…compatible; GPTBot/1.4; +https://openai.com/gptbot) → openai.com/gptbot.json. ChatGPT-User (…compatible; ChatGPT-User/1.0; +https://openai.com/bot) → openai.com/chatgpt-user.json. OAI-AdsBot → openai.com/adsbot.json. All four JSON endpoints return HTTP 200 (curl-verified). No mention of Web Bot Auth. Note Google-Extended has no UA at all, so it cannot be probed by request.

    How it scores

    Tier per evidence policy: scored — grade A meets the A/B bar required for scored audits.

    Example failure

    An article renders 6,200 chars of body text to Chrome. The same URL fetched as GPTBot/1.4 returns 200 with 480 chars — headline, dek, and a subscribe prompt — and the page’s JSON-LD NewsArticle omits isAccessibleForFree entirely. The site reads as cloaking to Google’s spam policy, and ChatGPT/Perplexity ingest and cite the 480-char stub as if it were the article.

    Sources