Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/no-bot-detection

    No aggressive bot-detection blocking agents

    What it checks

    Whether a bot-defense wall stopped the scan, and whether a challenge loader sits on a page that served no readable text. Bot-detection services such as Cloudflare Turnstile, DataDome and reCAPTCHA can block legitimate AI agents. Configure the service to allowlist the AI agents you want to admit.

    State Result
    the scan was rate-limited (HTTP 429) na
    a firewall or bot defense refused the scan fail, priority critical
    no wall detected and no page scanned na
    a challenge loader on a page that served no readable text warn, priority high
    no loader found, but no scanned page served readable text na
    a loader only on readable pages, or no loader at all pass

    Why it matters

    An edge bot-management rule that blocks or challenges requests carrying a named AI agent’s user-agent or IP hands that agent a non-200 or challenge response instead of the page. The content then cannot be fetched, indexed or cited. AI vendors publish per-agent IP ranges specifically so operators can allowlist them past those rules.

    Evidence

    • Cloudflare, Block AI bots. The feature blocks AI bots by behavior category: Search for content indexing, Agent for real-time automated activity, and Training (model development) — covering “Verified bots classified with that behavior, plus additional unverified bots that fall under these classifications”. From 15 September 2026 the platform default becomes: bots classified as Training or as Agent are blocked on pages that display ads, Search remains allowed — developers.cloudflare.com/…/block-ai-bots (verified 2026-08-21)
    • Cloudflare, “Content Independence Day” (1 July 2025): “Cloudflare, along with a majority of the world’s leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for their content.” Establishes that default-deny at the edge, not site-authored robots.txt, is now the dominant blocking mechanism — blog.cloudflare.com/content-independence-day-no-ai-… (verified 2026-08-21)
    • Perplexity crawler docs give operators explicit Cloudflare and AWS WAF allowlisting guidance and recommend combining “both User-Agent and IP address conditions”, with published endpoints https://www.perplexity.com/perplexitybot.json and https://www.perplexity.com/perplexity-user.json — docs.perplexity.ai/…/bots (verified 2026-08-21)
    • OpenAI publishes per-agent IP ranges (openai.com/searchbot.json, openai.com/gptbot.json, openai.com/chatgpt-user.json, openai.com/adsbot.json) for exactly this allowlisting purpose — developers.openai.com/…/bots (verified 2026-08-21)
    • Anthropic publishes verified crawler IPs at https://claude.com/crawling/bots.json and warns that “blocking IP address(es)…may not work correctly or persistently guarantee an opt-out” — support.claude.com/…/8896518-does-anthropic-crawl-d… (verified 2026-08-21)

    Counter-evidence: The A grade attaches to the ctx.wafProtection.isBlocked branch — observed blocking of the actual fetch. The second branch scans page HTML for the substrings recaptcha, challenges.cloudflare.com, hcaptcha.com and datadome.co. It has no documented consumer link. None of the vendor docs above state that the presence of a CAPTCHA widget on a page affects crawler access — and the page was by definition retrievable, since the scanner parsed it. Graded on its own, that sub-signal is D — presence of a form-scoped challenge widget is not evidence of agent blocking.

    Note also that the dominant 2026 mechanism the Cloudflare docs describe is UA/IP classification at the edge, which is invisible to a same-UA page scan — so the audit’s proven half is the half the implementation barely exercises.

    Limits

    A throttled scan proves nothing. HTTP 429 means “too many requests” — a statement about the rate this scan asked at, not about who the site admits. The audit returns not-applicable on a 429 rather than reporting a bot defense, and waf-detector.ts diagnoses it as a rate limit before it considers any provider, because every provider serves 429 for throttling. Until 2026-08-25 the audit failed at critical priority on a 429, which told storefronts that serve GPTBot perfectly well that their firewall blocks AI crawlers. Re-running the scan after a pause is the only way to get a verdict from a rate-limited origin.

    How it scores

    The blocking behavior is a documented product feature of the largest CDN, and multiple AI vendors document per-agent IP endpoints plus explicit WAF allowlisting instructions, which only make sense because bot defense demonstrably intercepts their agents.

    Sources