Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/ai-usage-signal-coherence-across-channels

    AI usage signals agree across every channel that carries them

    What it checks

    Normalises every AI-usage signal the site emits — robots.txt Allow/Disallow, AIPREF Content-Usage, legacy Content-Signal, TDMRep in its three transports, RSL permits/prohibits, and noai robots directives — into one comparable model and reports where they contradict each other. Different crawlers read different channels, so contradictory signals mean different AI systems reach opposite conclusions about the same content.

    Why it matters

    There is no defined precedence between these standards; each specifies only its own parsing. The channels differ:

    • TDMRep is carried in a well-known JSON array, an HTTP header and a meta tag (s17).
    • AIPREF is carried in robots.txt and an HTTP header (s11).
    • RSL is carried in robots.txt, a Link header, an HTML link and an inline script (s12).
    • Content-Signal is carried in robots.txt (s8).

    Falsifiable. Normalise each channel to (path-scope, usage-category, allow|deny) triples, then compare. Two channels asserting opposite values for the same category and overlapping path scope is a mechanically detectable contradiction. It provably yields divergent outcomes, because a TDMRep-aware crawler and an AIPREF-aware crawler read disjoint inputs. The highest-value instance is documented directly. Cloudflare’s managed robots.txt prepends Content-signal: search=yes, ai-train=no, use=reference above the operator’s own file, so the operator’s stated policy can be contradicted at the edge without their knowledge (s7).

    Evidence

    • Managed robots.txt — Cloudflare Bots — Cloudflare (vendor-doc, URL verified 2026-08-20)
    • Cloudflare prepends its own block to the origin’s robots.txt: User-Agent: * / Content-signal: search=yes, ai-train=no, use=reference / Allow: /. When the origin already serves robots.txt it combines both files, Cloudflare’s block first. This is the mechanism by which a site’s own AI policy gets silently overridden at the edge — directly auditable by diffing declared vs. served robots.txt.
    • Content Signals Policy (announcement) — Cloudflare (vendor-doc, URL verified 2026-08-20)
    • Defines the Content-Signal: robots.txt directive with signals search, ai-input, ai-train and values yes/no (omission = no preference). Canonical example: User-Agent: * / Content-Signal: search=yes, ai-train=no / Allow: /. Deployed as default on ~3.8M Cloudflare domains. Cloudflare deliberately does not emit an ai-input signal.
    • A Vocabulary For Expressing AI Usage Preferences (draft-ietf-aipref-vocab-07) — IETF aipref WG (Paul Keller, Open Future; Martin Thomson, Mozilla) (draft-spec, URL verified 2026-08-20)
    • ACTIVE, version 07, 2026-08-19, intended status Proposed Standard, WG-adopted. Categories: train-ai (modify learned parameters of a generative model) and search (select assets and direct users to their location, with excerpt conditions). Values are single-character tokens: y = allow, n = disallow, absent = unknown. Expressed as an RFC 8941 Structured Field dictionary, e.g. train-ai=y, search=n. Three-value outcome model: allowed / disallowed / unknown.
    • Attaching AI Usage Preferences to Content (draft-ietf-aipref-attach-05) — IETF aipref WG (draft-spec, URL verified 2026-08-20)
    • ACTIVE, version 05, 2026-08-18. Two attachment mechanisms: (1) HTTP response header Content-Usage: train-ai=n; (2) robots.txt directive Content-Usage: train-ai=n with optional path prefix, e.g. Content-Usage: /ai-ok/ train-ai=y, using the same path-prefix matching as Allow/Disallow. Precedence rules: preferences apply only to crawlable resources — “Disallowed paths have no associated usage preferences”; longest matching path prefix wins. No well-known location is defined.
    • RSL 1.0 Standard Specification — RSL Collective (spec, URL verified 2026-08-20)
    • robots.txt directive License: https://example.com/license.xml — “The value MUST be an absolute URI”; may be global or inside a User-agent group; multiple allowed. HTTP discovery: Link: <https://example.com/license.xml>; rel="license"; type="application/rsl+xml". HTML: <link rel="license" type="application/rsl+xml" href="..."> or inline <script type="application/rsl+xml">. no default/well-known location is mandated. XML: root <rsl xmlns="https://rslstandard.org/rsl" max-age>, <content url required, server, encrypted>, <license>, <permits|prohibits type="usage|user|geo">, <payment type="purchase|subscription|crawl|use|attribution|free">, <amount currency=ISO4217>, <standard>, <copyright type contactEmail contactUrl>, <legal type="warranty|disclaimer|attestation|contact|proof">.
    • TDM Reservation Protocol (TDMRep) — W3C CG Final Report — W3C Community Group (spec, URL verified 2026-08-20)
    • Four techniques, not three: (1) /.well-known/tdmrep.json — an ARRAY of objects each with location, tdm-reservation, tdm-policy; (2) HTTP response headers tdm-reservation: 1 and tdm-policy: <url>; (3) HTML <meta name="tdm-reservation" content="1"> / <meta name="tdm-policy" ...>; (4) EPUB tdm:reservation/tdm:policy. Note the well-known file is an array of rules — a bare object is non-conformant.

    How it scores

    Tier per evidence policy: scored — grade B meets the A/B bar required for scored audits.

    Example failure

    A publisher adds <meta name="tdm-reservation" content="1"> site-wide to claim an EU DSM Art. 4 opt-out, while robots.txt (unchanged for years) still carries User-agent: GPTBot / Allow: / with no Content-Usage directive, and Cloudflare has prepended Content-signal: search=yes, ai-train=no, use=reference. Three channels, three different answers for train-ai. A TDMRep-aware crawler skips the site, an AIPREF-aware crawler reads Cloudflare’s block rather than the publisher’s intent, and a robots-only crawler trains on everything.

    Sources