Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/ai-crawler-edge-parity

    AI crawlers get the same response from the edge that browsers get

    What it checks

    Detects the single most common and most invisible AI-visibility failure: robots.txt grants an AI crawler access, but the CDN/WAF in front of the origin answers that crawler with a challenge, a 403, a 402, or a proof-of-work interstitial. The site owner reads their own robots.txt and believes they are open; the crawler never sees a byte.

    Why it matters

    robots.txt (RFC 9309) is advisory metadata parsed by the crawler; the edge access decision is enforced independently by the WAF. Therefore a site can simultaneously publish User-agent: PerplexityBot / Allow: / and return a non-200 to every request carrying that user-agent. Falsifiable: fetch URL U with a browser UA and with crawler UA C; if robots.txt permits C for U and the C-request status is not 2xx while the browser-request is 200, the two policy layers contradict each other. Cloudflare makes one branch deterministically classifiable: a challenge response always carries cf-mitigated: challenge and content-type: text/html (s14).

    Evidence

    • Detect a Challenge Page response (cf-mitigated) — Cloudflare (vendor-doc, URL verified 2026-08-20)
    • Exact header is cf-mitigated; “challenge is the only valid value”; “The header is set for all Challenge Page types.” Also: “regardless of the requested resource-type, the content-type of a challenge will be text/html”. Gives a deterministic, vendor-documented way for an auditor to distinguish “you were challenged” from “you were served content” or “you hit an ordinary 403”.
    • OpenAI crawlers and user agents — OpenAI (vendor-doc, URL verified 2026-08-20)
    • Exact UA strings and published IP-range JSONs. OAI-SearchBot (…compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot) → openai.com/searchbot.json. GPTBot (…compatible; GPTBot/1.4; +https://openai.com/gptbot) → openai.com/gptbot.json. ChatGPT-User (…compatible; ChatGPT-User/1.0; +https://openai.com/bot) → openai.com/chatgpt-user.json. OAI-AdsBot → openai.com/adsbot.json. All four JSON endpoints return HTTP 200 (curl-verified). No mention of Web Bot Auth. Note Google-Extended has no UA at all, so it cannot be probed by request.
    • Verified bots policy — Cloudflare — Cloudflare (vendor-doc, URL verified 2026-08-20)
    • Two requirements for verified status: (1) “Honest self-identification — it declares who it is deterministically, through a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS.” (2) “Non-abusive behavior — it obeys robots.txt and crawl directives…”. Establishes that UA-string alone is never trusted, which is the source of the false-positive ambiguity when auditing edge blocks by UA spoofing.
    • Anubis (proof-of-work interstitial) — Techaro (vendor-doc, URL verified 2026-08-20)
    • The URL resolves, but served an Anubis deny page rather than the doc. That page itself confirmed the fingerprints an auditor needs: asset paths under /.within.website/x/cmd/anubis/, body text “Access Denied: error code <hex>”, the footer “Protected by Anubis From Techaro”, and a version banner (v1.27.1-…). Treat as a fingerprint source, not as a cited spec; re-verify the doc URL before shipping it in remediation copy.
    • AI Crawl Control — Cloudflare — Cloudflare (vendor-doc, URL verified 2026-08-20)
    • “Works automatically on all Cloudflare plans”; operators “Set allow or block rules for individual crawlers” and monitor AI access. Default-block posture and blocked-crawler response codes are not documented on this page — so an auditor must not assert Cloudflare’s default behaviour, only measure the observed response.

    How it scores

    Tier per evidence policy: scored — grade A meets the A/B bar required for scored audits.

    Example failure

    robots.txt contains User-agent: PerplexityBot / Allow: /. GET /pricing with a Chrome UA returns 200 and 4,100 chars of main text; the identical request with PerplexityBot returns 403 with cf-mitigated: challenge and content-type: text/html. Perplexity can never quote the pricing page, and nothing in the site’s own configuration reveals this.

    Sources