Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/machine-actionable-402-paid-access

    A 402 tells a crawler how to pay

    What it checks

    When a site charges for crawler access, this verifies the 402 response carries a price a machine can read and act on, in one of the three deployed formats. A 402 with a human HTML error page is functionally a hard block: the crawler cannot compute an offer, so it drops the URL rather than paying.

    Why it matters

    Every deployed paid-crawl scheme puts the price in a machine-readable slot, never in the body prose. Cloudflare pay-per-crawl returns 402 with crawler-price: USD XX.XX. The crawler then retries with crawler-exact-price, or pre-declares crawler-max-price, and receives 200 plus crawler-charged on success (s5). x402 v2 works differently. It puts a base64-encoded PaymentRequired payload in the PAYMENT-REQUIRED response header, carrying x402Version: 2 and an accepts[] array of {scheme, network, amount, asset, payTo, maxTimeoutSeconds, extra} (s19). RSL expresses it declaratively as <payment type="crawl"><amount currency="USD">0.015</amount></payment> (s12). Falsifiable: a 402 carrying none of these three signals contains no price any client can parse, so no retry is constructible.

    Evidence

    • Introducing pay per crawl: Enabling content owners to charge AI crawlers — Cloudflare (vendor-doc, URL verified 2026-08-20)
    • Exact wire format for paid crawling. Server returns HTTP 402 Payment Required with crawler-price: USD XX.XX. Crawler retries with crawler-exact-price: USD XX.XX, or pre-declares with crawler-max-price: USD XX.XX. Successful paid fetch returns 200 with crawler-charged: USD XX.XX.
    • Pay per crawl — AI Crawl Control — Cloudflare (vendor-doc, URL verified 2026-08-20)
    • Feature landing page confirming pay-per-crawl ships as a product surface under AI Crawl Control (private beta). Technical detail lives in the blog post (s5); this is the canonical product doc URL to cite in remediation text.
    • x402 Specification v2 — x402 Foundation (spec, URL verified 2026-08-20)
    • PaymentRequired payload carries "x402Version": 2 and an accepts array whose items have scheme, network, amount, asset, payTo, maxTimeoutSeconds, extra. “For HTTP, the canonical wire location is the base64-encoded PAYMENT-REQUIRED response header”; a PAYMENT-SIGNATURE header appears in examples. Protocol is transport-agnostic; HTTP binding lives in specs/transports-v2/http.md. Note the older repo path coinbase/x402 and specs/x402-specification.md are now 404.
    • RSL 1.0 Standard Specification — RSL Collective (spec, URL verified 2026-08-20)
    • robots.txt directive License: https://example.com/license.xml — “The value MUST be an absolute URI”; may be global or inside a User-agent group; multiple allowed. HTTP discovery: Link: <https://example.com/license.xml>; rel="license"; type="application/rsl+xml". HTML: <link rel="license" type="application/rsl+xml" href="..."> or inline <script type="application/rsl+xml">. no default/well-known location is mandated. XML: root <rsl xmlns="https://rslstandard.org/rsl" max-age>, <content url required, server, encrypted>, <license>, <permits|prohibits type="usage|user|geo">, <payment type="purchase|subscription|crawl|use|attribution|free">, <amount currency=ISO4217>, <standard>, <copyright type contactEmail contactUrl>, <legal type="warranty|disclaimer|attestation|contact|proof">.

    Limits

    • Every probe leaves the scanner’s own address with an AI crawler’s name in the User-Agent. It shows how the site answers a request that names the AI crawler. It does not show what the real crawler receives.
    • Real bots are told apart by network or signature, not by the string. Cloudflare grants verified status to a bot that “declares who it is deterministically, through a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS” (Cloudflare verified bots policy (verified 2026-10-19)). OpenAI publishes one IP-range file per bot, for example openai.com/gptbot.json and openai.com/chatgpt-user.json (OpenAI crawlers and user agents (verified 2026-10-19)). Google, whose crawlers this audit does not probe, illustrates the same rule: it documents a reverse-DNS check, or a match against its published IP ranges (Verifying Google crawlers and fetchers (verified 2026-10-19)). The scanner can present none of these.
    • The probe carries no Web Bot Auth signature, so it can only read the challenge. Cloudflare answers pay-per-crawl content with a 402 and says “To access this content, you must verify your AI crawler” (Cloudflare pay per crawl: Verify your AI crawler (verified 2026-10-19)), and that payment headers “must be included in the signature-input header components” (Cloudflare pay per crawl: Crawl pages (verified 2026-10-19)). The audit grades the 402 and its price headers. It never takes the paid path.
    • A 402 to a spoofed User-Agent is still a 402 the site chose to send. What a verified crawler would be offered on the same URL is not observed.

    How it scores

    Tier per evidence policy: scored — grade B meets the A/B bar required for scored audits.

    Example finding

    A publisher enables a paid-crawl rule at their edge, but the rule is a generic custom response: HTTP/1.1 402 Payment Required, content-type: text/html, body ‘Access to this content requires a licence. Contact sales@example.com.’ No crawler-price, no PAYMENT-REQUIRED, no RSL link. Every crawler willing to pay treats it as an unrecoverable error and removes the URL from its frontier; the publisher earns nothing and loses the citation.

    Sources