Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/web-bot-auth-request-tolerance

    A signed agent request is not rejected for being signed

    What it checks

    Probes whether the site’s edge and origin tolerate requests carrying RFC 9421 HTTP Message Signature headers at all. Some WAFs and origins reject unknown or oversized request headers outright, which means the entire cryptographically-verifiable-bot ecosystem — the one direction both Cloudflare and Google are building toward — cannot reach the site even when the operator wants it to.

    Why it matters

    Web Bot Auth signs outbound requests with three headers: Signature-Input (with tag="web-bot-auth", keyid, created, expires, nonce, alg), Signature, and Signature-Agent pointing at a JWKS directory at /.well-known/http-message-signatures-directory (s1, s3). Cloudflare’s verified-bot policy lists ‘a cryptographic Web Bot Auth signature’ as a first-class self-identification method (s4). Falsifiable claim: adding well-formed signature headers to an otherwise identical request must not change the response adversely. An adverse change is a 400 — the draft’s own malformed-header code — or a 403, a 421, or a 431 Request Header Fields Too Large.

    Where one occurs, the origin path cannot receive signed traffic, and no signed agent can ever be admitted, whoever signed it. The test is tolerance, not acceptance: the site is not expected to validate the auditor’s key.

    Evidence

    • HTTP Message Signatures for Automated Traffic Protocol (draft-meunier-webbotauth-httpsig-protocol-02) — IETF / Thibault Meunier (Cloudflare), Sandor Major (Google) (draft-spec, URL verified 2026-08-20)
    • ACTIVE draft-02, last updated 2026-08-18. Defines three request headers: Signature, Signature-Input (params: created, expires, keyid, tag), and Signature-Agent (Structured Dictionary of HTTPS URLs, default type="directory"). Defines well-known URI /.well-known/http-message-signatures-directory serving a JWKS with media type application/http-message-signatures-directory+json. Origin MUST parse the three headers, resolve Signature-Agent, validate; MAY return 400 Bad Request on malformed headers and 403 Forbidden when additional signatures are required.
    • Web Bot Auth — Cloudflare Bots docs — Cloudflare (vendor-doc, URL verified 2026-08-20)
    • Confirms the deployed shape of the three headers: Signature-Input carries tag="web-bot-auth", keyid, created/expires, nonce, alg; Signature-Agent is a structured string such as “signature-agent.test”. Bots must host a JWKS at /.well-known/http-message-signatures-directory over HTTPS. Cloudflare verifies server-side against its registered bot database; site operators configure nothing themselves — which is exactly why a site can silently reject signed traffic without knowing.
    • Verified bots policy — Cloudflare — Cloudflare (vendor-doc, URL verified 2026-08-20)
    • Two requirements for verified status: (1) “Honest self-identification — it declares who it is deterministically, through a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS.” (2) “Non-abusive behavior — it obeys robots.txt and crawl directives…”. Establishes that UA-string alone is never trusted, which is the source of the false-positive ambiguity when auditing edge blocks by UA spoofing.
    • RFC 9421: HTTP Message Signatures — IETF (spec, URL verified 2026-08-20)
    • Ratified standard underlying Web Bot Auth. §4.1 defines Signature-Input, §4.2 defines Signature.
    • §2.3 defines the tag parameter: “An application-specific tag for the signature as a String value… used by applications to help identify signatures relevant for specific applications or protocols”. web-bot-auth uses tag="web-bot-auth".
    • §5.1 defines the Accept-Signature response field, for negotiating signatures in subsequent messages. It is the only standardised way an origin can advertise that it wants signed requests.
    • Web Bot Auth Architecture (draft-meunier-web-bot-auth-architecture-05) — IETF (draft-spec, URL verified 2026-08-20)
    • Version 05, last updated 2026-03-02, now EXPIRED and replaced by draft-meunier-webbotauth-httpsig-protocol. Cite s1, not this, for current header semantics; useful only as the architectural rationale (identifying automated traffic via HTTP Message Signatures).

    How it scores

    Tier per evidence policy: scored — grade B meets the A/B bar required for scored audits.

    Example failure

    A site fronted by a strict WAF rule that rejects requests with unrecognised headers returns 200 to the baseline GET and 403 to the identical GET carrying Signature, Signature-Input and Signature-Agent. Every Web Bot Auth-signed agent — precisely the well-behaved, cryptographically identifiable population the operator would want to allow — is rejected before any verification can occur, and nothing in the site’s config names this as an AI-access decision.

    Sources