Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/https-enabled

    HTTPS enabled

    What it checks

    Enterprise AI frameworks refuse to interact with non-HTTPS sites due to security policies. GPTBot, ClaudeBot, and enterprise RAG systems all skip HTTP-only sites entirely, making your content invisible to AI-generated answers. Enable HTTPS with a valid TLS certificate.

    Why it matters

    HTTPS requirement — Serving the site over HTTPS with a valid certificate is a precondition for AI-agent surfaces to retrieve or act on the site. Agent-protocol specs mandate HTTPS outright. Well-known agent and security files are defined as HTTPS-only. Browser-based agents inherit Chromium’s mixed-content and HTTPS-First behaviour.

    WAF — Edge bot-management sitting in front of the origin can deny AI crawlers and agents outright, or gate them behind payment, so a site with perfect content structure can still be entirely invisible to AI systems.

    Evidence

    HTTPS requirement

    Strongest evidence is from agent-adjacent ratified/near-ratified specs rather than crawler docs. MCP (2025-11-25) states plainly: ‘All authorization server endpoints MUST be served over HTTPS’, and ‘All redirect URIs MUST be either localhost or use HTTPS’. Client ID Metadata Documents are required at HTTPS URLs. Any MCP-exposed site capability is therefore unreachable without TLS. RFC 9116 requires security.txt to be ‘accessed exclusively via HTTPS’. Browser-resident agents (ChatGPT Atlas, Comet, Gemini-in-Chrome, Claude in Chrome) run on Chromium and therefore inherit mixed-content blocking and HTTPS-First warnings, so an HTTP-only page degrades for the fastest-growing agent class. Google’s AI-features eligibility runs through normal Search indexing, where HTTPS has been a documented positive signal since 2014.

    WAF

    Fully documented vendor behaviour at large scale. Cloudflare began blocking AI crawlers by default for new domains on 1 July 2025, making permission the default posture across a very large share of the web. It has since announced that from 15 September 2026 new domains will block ‘Training’ and ‘Agent’ class bots by default on ad-displaying pages while leaving ‘Search’ allowed, and has extended managed robots.txt with a use signal (immediate / reference / full).

    Pay-per-crawl operationalises the gate in HTTP. Crawlers ‘either present payment intent via request headers for successful HTTP 200 access, or receive an HTTP 402 Payment Required response with pricing’, at a minimum of $0.001 per crawl. Critically, an existing WAF or Bot Management block rule overrides pay-per-crawl’s charge behaviour, and silently converts a monetizable crawl into a hard block. Adoption context: of 3,816 top-10k domains with robots.txt, ~14% carried AI-bot directives; GPTBot was the most-disallowed at 312 domains.

    Verification is moving to cryptography — Cloudflare’s Web Bot Auth implements RFC 9421 HTTP Message Signatures with Signature-Input / Signature / Signature-Agent headers and a JWKS key directory at /.well-known/http-message-signatures-directory.

    Limits

    HTTPS requirement — No AI crawler vendor documents HTTPS as a requirement. developers.openai.com/api/docs/bots, Anthropic’s crawler support article and docs.perplexity.ai all say nothing about TLS; GPTBot, ClaudeBot and PerplexityBot are ordinary HTTP clients and will fetch http:// URLs. So the strict claim ‘AI crawlers refuse HTTP’ is unproven and should not be asserted. The honest claim is narrower: HTTPS is mandatory for agent protocols and browser-based agents, and is a universal baseline (>95% of page loads) such that its absence is a strong negative quality signal. HSTS specifically has no documented AI consumer at all — see the security-headers signal.

    WAF — Two caveats that shape how this must be audited. (1) User-Agent-based probing is unreliable in both directions: Cloudflare documented Perplexity using a Chrome-impersonating stealth crawler on unlisted IPs across rotating ASNs at 3–6M requests/day to evade no-crawl directives, and conversely malicious scrapers routinely spoof GPTBot. An audit that merely sets a UA header measures the WAF’s UA rules, not real agent access — genuine verification requires the vendors’ published IP ranges (openai.com/gptbot.json, claude.com/crawling/bots.json, perplexity.com/perplexitybot.json) or Web Bot Auth signatures. (2) Blocking is often a deliberate, rational business decision, not a defect.

    Cloudflare’s crawl-to-refer data puts Anthropic at about 71,000 crawls per HTML referral in the June 2025 window, with the caveat that Claude’s native app sends no Referer. That makes uncompensated crawl a real cost. The audit should REPORT the gate neutrally as ‘AI agents are blocked here’ rather than scoring it as a failure, since the site owner may have chosen it.

    How it scores

    HTTPS requirement — The evidence is strong but it comes from agent-adjacent specifications rather than from crawler documentation. MCP states that “All authorization server endpoints MUST be served over HTTPS”. The well-known agent and security files are defined HTTPS-only. Browser-based agents inherit Chromium’s mixed-content and HTTPS-First behaviour. An agent surface that requires HTTPS genuinely cannot be reached without it. What is missing for an A is a crawler vendor saying so: OpenAI’s, Anthropic’s and Perplexity’s crawler documents say nothing about TLS, and their crawlers are ordinary HTTP clients that will fetch http:// URLs. The strict claim “AI crawlers refuse HTTP” is unproven, and the audit does not make it.

    WAF — Cloudflare documents the behaviour at scale and by date. It began blocking AI crawlers by default for new domains on 1 July 2025. It has announced that from 15 September 2026, new domains on ad-displaying plans will block Training and Agent class bots by default. That is a named intermediary documenting exactly what it does to the traffic this audit is about, which is grade A. Two caveats shape how it can be measured, rather than whether it is true. User-agent probing is unreliable in both directions: Perplexity was documented using a Chrome-impersonating stealth crawler, and scrapers routinely spoof GPTBot. An opaque block therefore cannot be told apart from correct impersonation defence.

    Sources