Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/ai-usage-signal-coherence-across-channels

    AI usage signals agree across every channel that carries them

    What it checks

    Normalises every AI-usage signal the site emits — robots.txt Allow/Disallow, AIPREF Content-Usage, Content-Signal, TDMRep in its three transports, RSL permits/prohibits, and noai robots directives — into one comparable model and reports where they contradict each other. Different crawlers read different channels, so contradictory signals mean different AI systems reach opposite conclusions about the same content.

    Why it matters

    There is no defined precedence between these standards; each specifies only its own parsing. The channels differ:

    • TDMRep is carried in a well-known JSON array, an HTTP header and a meta tag (s17).
    • AIPREF is carried in robots.txt and an HTTP header (s11).
    • RSL is carried in robots.txt, a Link header, an HTML link and an inline script (s12).
    • Content-Signal is carried in robots.txt (s8).

    Falsifiable. Normalise each channel to (path-scope, usage-category, allow|deny) triples, then compare. Two channels asserting opposite values for the same category and overlapping path scope is a mechanically detectable contradiction. It provably yields divergent outcomes, because a TDMRep-aware crawler and an AIPREF-aware crawler read disjoint inputs. Content Signals comparisons retain explicit bot and path scope. Distinct access rules do not contradict use permission. Overlapping scopes receive review notes rather than guessed effective precedence.

    Evidence

    • Managed robots.txt — Cloudflare Bots — Cloudflare (vendor-doc, URL verified 2026-08-20)
    • Cloudflare prepends its own block to the origin’s robots.txt: User-Agent: * / Content-signal: search=yes, ai-train=no, use=reference / Allow: /. When the origin already serves robots.txt it combines both files, Cloudflare’s block first. Explicit managed-block markers support a qualified Cloudflare note. Line order alone does not establish ownership or policy precedence.
    • Content Signals Policy (announcement) — Cloudflare (vendor-doc, URL verified 2026-08-20)
    • Defines the Content-Signal: robots.txt directive with signals search, ai-input, ai-train and values yes/no (omission = no preference). Canonical example: User-Agent: * / Content-Signal: search=yes, ai-train=no / Allow: /. Deployed as default on ~3.8M Cloudflare domains. Cloudflare deliberately does not emit an ai-input signal.
    • A Vocabulary For Expressing AI Usage Preferences (draft-ietf-aipref-vocab-07) — IETF aipref WG (Paul Keller, Open Future; Martin Thomson, Mozilla) (draft-spec, URL verified 2026-08-20)
    • ACTIVE, version 07, 2026-08-19, intended status Proposed Standard, WG-adopted. Categories: train-ai (modify learned parameters of a generative model) and search (select assets and direct users to their location, with excerpt conditions). Values are single-character tokens: y = allow, n = disallow, absent = unknown. Expressed as an RFC 8941 Structured Field dictionary, e.g. train-ai=y, search=n. Three-value outcome model: allowed / disallowed / unknown.
    • Attaching AI Usage Preferences to Content (draft-ietf-aipref-attach-05) — IETF aipref WG (draft-spec, URL verified 2026-08-20)
    • ACTIVE, version 05, 2026-08-18. Two attachment mechanisms: (1) HTTP response header Content-Usage: train-ai=n; (2) robots.txt directive Content-Usage: train-ai=n with optional path prefix, e.g. Content-Usage: /ai-ok/ train-ai=y, using the same path-prefix matching as Allow/Disallow. Precedence rules: preferences apply only to crawlable resources — “Disallowed paths have no associated usage preferences”; longest matching path prefix wins. No well-known location is defined.
    • RSL 1.0 Standard Specification — RSL Collective (spec, URL verified 2026-08-20)
    • robots.txt directive License: https://example.com/license.xml — “The value MUST be an absolute URI”; may be global or inside a User-agent group; multiple allowed. HTTP discovery: Link: <https://example.com/license.xml>; rel="license"; type="application/rsl+xml". HTML: <link rel="license" type="application/rsl+xml" href="..."> or inline <script type="application/rsl+xml">. no default/well-known location is mandated. XML: root <rsl xmlns="https://rslstandard.org/rsl" max-age>, <content url required, server, encrypted>, <license>, <permits|prohibits type="usage|user|geo">, <payment type="purchase|subscription|crawl|use|attribution|free">, <amount currency=ISO4217>, <standard>, <copyright type contactEmail contactUrl>, <legal type="warranty|disclaimer|attestation|contact|proof">.
    • TDM Reservation Protocol (TDMRep) — W3C CG Final Report — W3C Community Group (spec, URL verified 2026-08-20)
    • Four techniques, not three: (1) /.well-known/tdmrep.json — an ARRAY of objects each with location, tdm-reservation, tdm-policy; (2) HTTP response headers tdm-reservation: 1 and tdm-policy: <url>; (3) HTML <meta name="tdm-reservation" content="1"> / <meta name="tdm-policy" ...>; (4) EPUB tdm:reservation/tdm:policy. Note the well-known file is an array of rules — a bare object is non-conformant.

    Limits

    Content Signals and access rules express different choices. A permitted use can have a crawl obstacle without a policy contradiction. Cross-format search equivalence is not established. Unknown extensions are retained without invented equivalence. No source establishes universal Content Signals consumption (contentsignals.org, verified 2026-10-25). Overlapping scopes need review; only identical scopes support a Content Signals contradiction.

    The comparison stores at most 50 review notes and reports the omitted count. Content Signals preflight limits prevent incomplete robots.txt evidence from supporting a silence or complete-agreement claim.

    How it scores

    Tier per evidence policy: scored — grade B meets the A/B bar required for scored audits.

    A site that publishes no AI-usage signal in any channel is not applicable: with nothing declared there is nothing to contradict. The AIPREF vocabulary reads an absent preference as unknown and Content Signals reads omission as no preference, so neither documents a cost for silence.

    Example finding

    A site publishes Content-Signal: ai-train=no and Content-Usage: train-ai=y for the same bot and path. The declared training preferences differ. The scan does not prove which policy an external crawler follows.

    Sources