Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/aipref-content-usage-declaration-validity

    AIPREF Content-Usage declarations are valid and can be read

    What it checks

    Checks whether the site expresses AI usage preferences in the IETF AIPREF form that standards-conformant crawlers will actually parse: Content-Usage in robots.txt, as an HTTP response header, or both. It then validates the syntax and the vocabulary, and checks whether the declared preferences are attached to paths where they legally have effect.

    Why it matters

    draft-ietf-aipref-attach-05 defines the only two attachment points a conformant crawler reads. They are the Content-Usage HTTP response header, and the Content-Usage: robots.txt directive with an optional path prefix (s11). draft-ietf-aipref-vocab-07 fixes the value grammar: an RFC 8941 structured-field dictionary over the categories train-ai and search, with token values y and n. Anything absent is unknown, and crawlers resolve unknown using their own default rather than yours (s10). Three falsifiable consequences follow.

    First, a site publishing only Cloudflare’s Content-Signal: search=yes, ai-train=no emits zero AIPREF preference: an AIPREF parser sees unknown for every category. Second, yes and no are not valid AIPREF tokens, so Content-Usage: train-ai=no fails structured-field parsing. Third, attach-05 states that ‘Disallowed paths have no associated usage preferences’, so a Content-Usage scoped to a path the same agent group Disallows is inert by specification.

    Evidence

    • A Vocabulary For Expressing AI Usage Preferences (draft-ietf-aipref-vocab-07) — IETF aipref WG (Paul Keller, Open Future; Martin Thomson, Mozilla) (draft-spec, URL verified 2026-08-20)
    • ACTIVE, version 07, 2026-08-19, intended status Proposed Standard, WG-adopted. Categories: train-ai (modify learned parameters of a generative model) and search (select assets and direct users to their location, with excerpt conditions). Values are single-character tokens: y = allow, n = disallow, absent = unknown. Expressed as an RFC 8941 Structured Field dictionary, e.g. train-ai=y, search=n. Three-value outcome model: allowed / disallowed / unknown.
    • Attaching AI Usage Preferences to Content (draft-ietf-aipref-attach-05) — IETF aipref WG (draft-spec, URL verified 2026-08-20)
    • ACTIVE, version 05, 2026-08-18. Two attachment mechanisms: (1) HTTP response header Content-Usage: train-ai=n; (2) robots.txt directive Content-Usage: train-ai=n with optional path prefix, e.g. Content-Usage: /ai-ok/ train-ai=y, using the same path-prefix matching as Allow/Disallow. Precedence rules: preferences apply only to crawlable resources — “Disallowed paths have no associated usage preferences”; longest matching path prefix wins. No well-known location is defined.
    • Content Signals Policy (announcement) — Cloudflare (vendor-doc, URL verified 2026-08-20)
    • Defines the Content-Signal: robots.txt directive with signals search, ai-input, ai-train and values yes/no (omission = no preference). Canonical example: User-Agent: * / Content-Signal: search=yes, ai-train=no / Allow: /. Deployed as default on ~3.8M Cloudflare domains. Cloudflare deliberately does not emit an ai-input signal.
    • Content Signals (verified 2026-10-25) — A separate optional publication format. Its absence from an AIPREF parser does not justify requiring AIPREF adoption.
    • Managed robots.txt — Cloudflare Bots — Cloudflare (vendor-doc, URL verified 2026-08-20)
    • Cloudflare prepends its own block to the origin’s robots.txt: User-Agent: * / Content-signal: search=yes, ai-train=no, use=reference / Allow: /. When the origin already serves robots.txt it combines both files, Cloudflare’s block first. Explicit managed-block markers support a qualified Cloudflare note. Line order alone does not establish ownership or policy precedence.

    Limits

    Content Signals is not superseded by AIPREF. Neither format requires adoption of the other. Managed block ownership cannot be inferred from line order.

    How it scores

    Tier per evidence policy: scored — grade B meets the A/B bar required for scored audits.

    Example finding

    Content-Usage: train-ai=no uses a Content-Signal value in a different format. AIPREF requires y or n. A site publishing only Content-Signal has no AIPREF declaration to validate and receives no warning for that choice.

    Sources