Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    operability-safety/security-header-hygiene

    security.txt (RFC 9116)

    What it checks

    One root file, one signal.

    State Result
    a 200 response whose body is not HTML, carrying a Contact field and an Expires date in the future pass
    a published file that returns HTML at 200 (SPA soft-404), or has no Contact, no Expires, an unparseable Expires, or an Expires that has passed warn, priority low
    no security.txt published (non-200 at both locations) na
    the location was never fetched, so nothing was measured na

    /.well-known/security.txt is the location checked; the legacy top-level /security.txt is accepted as a fallback and named as such in the result. Detection is parse-not-probe by design — this dossier’s own evidence records that only a minority of deployed files pass RFC validation, so presence alone proves nothing.

    fail is never returned, and scoreDisplayMode: 'informative' with weight: 0 keeps every outcome out of the category score, the readiness vitals and the top-fails list.

    Why it matters

    Falsifiable claim: none is made about AI agents. /.well-known/security.txt is a vulnerability-disclosure file whose stated consumers are human security researchers and vulnerability-notification tooling. The evidence review found no AI crawler, retrieval pipeline or answer engine documented to read it, so the audit makes no claim that publishing one changes AI-agent behaviour. It reports the file’s RFC 9116 conformance at weight 0, and never fails the site.

    What the audit does claim, and what is testable, is narrower. RFC 9116 defines what a security.txt must contain. A published file with no Contact, no Expires, an unparseable Expires, or an Expires in the past does not conform — and it advertises a disclosure route that no longer works. That claim is about the file, not about agents, which is why the tier is informative.

    Evidence

    Security headers

    Mechanism: The claim under test: the presence of HSTS / CSP / X-Content-Type-Options / Referrer-Policy / Permissions-Policy response headers changes whether or how an AI crawler or agent retrieves, parses, trusts or cites the page.

    No supporting evidence was found. An exhaustive read of the AI crawler documentation from OpenAI, Anthropic, Perplexity, Apple and Google turned up not a single reference to any of these headers. Google’s AI-features guidance goes further and states there are ‘no additional technical requirements’ for AI Overviews / AI Mode beyond ordinary Search snippet eligibility. Cloudflare’s AI Crawl Control — the product that actually sits between AI crawlers and origins — makes decisions on user agent, IP, signature and robots.txt, never on the origin’s security headers.

    HTTPS requirement

    Mechanism: Serving the site over HTTPS with a valid certificate is a precondition for AI-agent surfaces to retrieve or act on the site. Agent-protocol specs mandate HTTPS outright. Well-known agent and security files are defined as HTTPS-only. Browser-based agents inherit Chromium’s mixed-content and HTTPS-First behaviour.

    MCP (2025-11-25) states plainly: ‘All authorization server endpoints MUST be served over HTTPS’ and ‘All redirect URIs MUST be either localhost or use HTTPS’. RFC 9116 requires security.txt to be ‘accessed exclusively via HTTPS’. Browser-resident agents (ChatGPT Atlas, Comet, Gemini-in-Chrome, Claude in Chrome) run on Chromium and inherit mixed-content blocking, so an HTTP-only page degrades for the fastest-growing agent class.

    security.txt

    Mechanism: The claim under test: AI agents read /.well-known/security.txt to identify the site operator or a disclosure contact, and its presence improves how agents treat the site.

    security.txt is a real, published IETF document (RFC 9116). It has a well-defined location, a media type (text/plain, UTF-8, HTTPS-only) and required fields (Contact, Expires). Adoption is genuine, if small: roughly 0.7% of the top 1M domains in April 2024, rising to about 1.25% in 2025, with a broader count of about 573,000 domains by 2026.

    Correct Content-Type for llms.txt and .md files

    Carried here only to record where the nosniff sub-signal went. X-Content-Type-Options: nosniff removes a browser’s ability to recover from a wrong Content-Type. What an agent needs is the correct type. machine-discovery/ai-file-delivery (v1 8.10) measures that on the AI files themselves, and names nosniff when a mis-typed file carries it. This audit does not read nosniff, and no audit checks it on the homepage.

    Limits

    Security headers — These are browser-enforced defence-in-depth mechanisms with human users and browsers as their consumers; server-side crawlers do not implement any of them. The only genuine adjacencies run in the opposite direction from the v1 audits. First, CSP frame-ancestors and X-Frame-Options can stop a page being embedded in an agent surface, so a strict policy is an agent-readiness negative rather than a positive. Second, OpenAI’s Apps SDK shows the agent host imposing CSP on its own widget iframe, which is a property of the app and not of the publisher’s site. Third, X-Content-Type-Options: nosniff matters only in a browser, and only makes a wrong Content-Type more fatal — so it belongs to the content-type signal, not here.

    HTTPS requirement — No AI crawler vendor documents HTTPS as a requirement, and HSTS specifically has no documented AI consumer at all — the header is a browser-state mechanism layered on top of the TLS the agents actually need. This is why the B grade lives on the transport signal while the audit that reports the header stays informative. The scored HTTPS check itself is access-crawl-control/https-enabled (v1 8.1), which this audit does not duplicate.

    security.txt — RFC 9116 is INFORMATIONAL, explicitly ‘not an Internet Standards Track specification’. Its stated consumers are human security researchers and vulnerability-notification tooling; no AI vendor documentation mentions security.txt at all. Conformity is poor — analyses find only a minority of deployed files pass RFC validation, so presence alone is weak evidence of anything. Hence the parse-not-probe detection in this audit, and the informative tier.

    How it scores

    Grade C, tier informative, weight 0. The grade comes from the one signal the audit measures: security.txt.

    • RFC 9116 is a real IETF document with a defined location and required fields. That earns more than D.
    • RFC 9116 is Informational, adoption is small, and no AI vendor documents reading the file. That keeps it below B.
    • Grade C maps to the informative tier, so weightForGrade returns 0. No outcome moves a score.
    • The audit never fails. A conforming file passes. A published but non-conforming file warns at low priority. An absent file is not applicable.

    The audit once shipped at B. That B belonged to the HTTPS signal below, which this audit never measured. HTTPS ships scored as access-crawl-control/https-enabled.

    Sources