Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/tdm-rep

    TDM-Rep declaration

    What it checks

    Whether your site publishes a TDM-Rep declaration — a machine-readable text-and-data-mining reservation — and, if so, in which direction it points.

    The audit reads three channels, in this order, and reports the first one that carries a declaration:

    1. a tdm-reservation response header (with tdm-policy), which the specification calls “currently the preferred technique”;
    2. a /.well-known/tdmrep.json policy file, validated against the shape the specification defines: an array of objects, each carrying location and tdm-reservation, optionally tdm-policy;
    3. a <meta name="tdm-reservation"> tag, collected across every scanned page.

    The file is read only when it is served as JSON or plain text and does not start with <, so an HTML catch-all page is not mistaken for a policy. A JSON document that names no TDM-Rep key at all is somebody else’s file, such as one document a server returns for every /.well-known/* path, and also counts as no file.

    The two directions are reported as distinct outcomes, never as the same one: tdm-reservation="1" reserves your mining rights, "0" permits mining. Any other value is a protocol error and warns. A file that does not parse, a file with the wrong shape, pages that disagree, and a file with different reservations per location also warn. The audit never fails. A site that publishes no declaration is not applicable, not a failure — nothing is documented to read it, so its absence is not a defect.

    Nothing here changes your score. TDM-Rep is a W3C Community Group Final Report, explicitly not a W3C Standard, and its value is legal evidence of an opt-out rather than a change in any agent’s behaviour.

    Why it matters

    An AI crawler fetches /.well-known/tdmrep.json or reads <meta name="tdm-reservation"> and changes whether it collects or uses the page’s content as a result.

    Evidence

    • W3C TDM Reservation Protocol, Community Group Final Report (10 May 2024). Its own status section says: “It is not a W3C Standard nor is it on the W3C Standards Track.” It defines exactly the three signalling methods the audit looks for. A /.well-known/tdmrep.json well-known file. A tdm-reservation HTTP response header, described in the report as “currently the preferred technique”. And <meta name="tdm-reservation"> or <meta name="tdm-policy"> in HTML — w3.org/…/CG-FINAL-tdmrep-20240510 (verified 2026-10-17)
    • Spec-defined file shape: “This file contains an array of JSON objects”. Each object requires location and tdm-reservation, and tdm-policy is optional. The array is required even for a single rule. For tdm-reservation, 1 reserves the rights, 0 does not, and “Other values are considered protocol errors”. The audit validates the file against this shape and warns on anything else (same URL, verified 2026-10-17)
    • IANA lists tdmrep.json in the Well-Known URIs registry, status provisional, change controller W3C, registered 2024-02-15 — iana.org/…/well-known-uris-1.csv (verified 2026-10-18). Registration reserves the path; it does not make anyone read it.
    • One consumer reads the header. Spawning’s datadiligence library (“Respect generative AI opt-outs in your ML training pipeline”) reads the tdm-reservation HTTP header only and treats 1 as disallowed; it does not read the file or the meta tag — github.com/…/http.py (verified 2026-10-18)
    • Adoption is publisher-side, not crawler-side. The CG names Mondadori, Penguin Random House, the STM association, Copyright Clearance Center, Taylor & Francis and the BBC among participants. It records that “Spawning AI has already integrated partially the opt-out solution developed by the TDM Rep CG in their service” — the only named consuming implementer found — w3.org/…/tdmrep (verified 2026-08-21)

    Limits

    No major AI vendor documents honoring the protocol. OpenAI’s crawler documentation describes robots.txt and published IP ranges only, with no mention of TDM signals (developers.openai.com/…/bots); Anthropic’s crawler article describes robots.txt directives and Crawl-delay only (support.claude.com/…/8896518-does-anthropic-crawl-d…); Perplexity’s documents robots.txt and WAF allowlisting only (docs.perplexity.ai/…/bots) — all verified 2026-08-21. Standardization momentum has also moved elsewhere. The IETF AIPREF working group is chartered to standardize AI-preference expression via “Well-Known URIs (RFC 8615) such as the Robots Exclusion Protocol (RFC 9309), and HTTP response header fields”, with IESG submission targeted for 31 August 2026.

    Its charter does not reference TDM-Rep — datatracker.ietf.org/…/about (verified 2026-08-21). Finally, the signal is directionally orthogonal to agent readiness: a reservation value of 1 denies mining and 0 permits it. The audit names the direction in its message and scores neither.

    How it scores

    TDM-Rep is a genuinely published specification with real publisher-side participation and one named partial implementer, but no major crawler operator documents consuming it, so the causal claim about agent behavior is plausible and unproven rather than demonstrated.

    Sources