Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/tdm-rep

    TDM-Rep declaration

    What it checks

    Whether your site publishes a TDM-Rep declaration — a machine-readable text-and-data-mining reservation — and, if so, in which direction it points.

    The declaration lives in one of two places: a <meta name="tdm-reservation"> tag, or a /.well-known/tdmrep.json policy file, which is validated against the shape the specification defines (an array of objects, each carrying location and tdm-reservation, optionally tdm-policy).

    The two directions are reported as distinct outcomes, never as the same one: tdm-reservation="1" reserves your mining rights, "0" permits mining. A site that publishes no declaration is not applicable, not a failure — nothing is documented to read the file, so its absence is not a defect.

    Nothing here changes your score. TDM-Rep is a W3C Community Group Final Report, explicitly not a W3C Standard, and its value is legal evidence of an opt-out rather than a change in any agent’s behaviour.

    Why it matters

    An AI crawler fetches /.well-known/tdmrep.json or reads <meta name="tdm-reservation"> and changes whether it collects or uses the page’s content as a result.

    Evidence

    • W3C TDM Reservation Protocol, Community Group Final Report (10 May 2024). Its own status section says: “It is not a W3C Standard nor is it on the W3C Standards Track.” It defines exactly the three signalling methods the audit looks for. A /.well-known/tdmrep.json well-known file. A tdm-reservation HTTP response header, described in the report as “currently the preferred technique”. And <meta name="tdm-reservation"> or <meta name="tdm-policy"> in HTML — w3.org/…/CG-FINAL-tdmrep-20240510 (verified 2026-08-21)
    • Spec-defined file shape: an array of objects, each with mandatory location and tdm-reservation and optional tdm-policy — so the audit’s Record<string, unknown> parse accepts non-conforming documents (same URL, verified 2026-08-21)
    • Adoption is publisher-side, not crawler-side. The CG names Mondadori, Penguin Random House, the STM association, Copyright Clearance Center, Taylor & Francis and the BBC among participants. It records that “Spawning AI has already integrated partially the opt-out solution developed by the TDM Rep CG in their service” — the only named consuming implementer found — w3.org/…/tdmrep (verified 2026-08-21)

    Limits

    No major AI vendor documents honoring the protocol. OpenAI’s crawler documentation describes robots.txt and published IP ranges only, with no mention of TDM signals (developers.openai.com/…/bots); Anthropic’s crawler article describes robots.txt directives and Crawl-delay only (support.claude.com/…/8896518-does-anthropic-crawl-d…); Perplexity’s documents robots.txt and WAF allowlisting only (docs.perplexity.ai/…/bots) — all verified 2026-08-21. Standardization momentum has also moved elsewhere. The IETF AIPREF working group is chartered to standardize AI-preference expression via “Well-Known URIs (RFC 8615) such as the Robots Exclusion Protocol (RFC 9309), and HTTP response header fields”, with IESG submission targeted for 31 August 2026.

    Its charter does not reference TDM-Rep — datatracker.ietf.org/…/about (verified 2026-08-21). Finally, the signal is directionally orthogonal to agent readiness: a reservation value of 1 denies mining, and the audit passes it identically to 0.

    How it scores

    TDM-Rep is a genuinely published specification with real publisher-side participation and one named partial implementer, but no major crawler operator documents consuming it, so the causal claim about agent behavior is plausible and unproven rather than demonstrated.

    Sources