Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    machine-discovery/llms-txt-exists

    llms.txt exists

    What it checks

    GET /llms.txt returns 200 with a markdown body, plus — reported, not scored — whether a <link> in the page head points at that file.

    State Result
    200 with a body starting with # pass
    200 but no markdown heading warn, priority high
    non-200 or no response fail, priority critical

    The discovery <link> is appended to found in every branch, as either discovery <link> → <href> or no discovery <link> in <head>. When the file is missing and a link points at it, the message says so. A link to a file that is not served is the one case where the absorbed signal changes what the user reads.

    Why it matters

    /llms.txt existence at domain root — Publishing a well-formed /llms.txt at the domain root causes major AI crawlers and agents (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended) to fetch it and use it to navigate the site, measurably increasing retrieval coverage or citation rate.

    llms.txt content quality conventions — An llms.txt that conforms to structural conventions is machine-parseable and passes automated validation, whereas a malformed one fails. Split precisely: (a) H1 present, (b) at least one text markdown link, (c) length >= 50 chars are enforced by a real shipping consumer; (d) blockquote summary, (e) per-link descriptions, (f) link validity are spec-optional with no known enforcing consumer.

    Soft-404 — A server that returns HTTP 200 with an application shell for URLs that do not exist causes two distinct harms. Crawlers spend capacity on valueless error pages, and may index them. And any automated audit that infers file existence from a 2xx status produces false positives — reporting llms.txt, robots.txt, feeds or .md mirrors as ‘present’ when the origin merely echoed the SPA shell.

    Evidence

    /llms.txt existence at domain root

    Refuted as a retrieval mechanism, and supported only as a signal of convention-following. The spec is real and actively maintained: llmstxt.org v2, 2026-08-10, about 2.6k GitHub stars. Adoption is non-trivial too — 8.7% of the Tranco top-1,000, or 15.8% of the 549 reachable, and about 10.13% across a 300k-domain SE Ranking sample. All four major AI labs publish one, with HTTP 200 verified on Anthropic, OpenAI, Perplexity, Cloudflare, Stripe, GitHub and Vercel.

    Google Chrome ships a real llms-txt audit in Lighthouse 13.3’s Agentic Browsing category — the audit and gatherer source were read directly. But no vendor anywhere documents an agent consuming it. Four independent log studies converge on near-zero AI-crawler interest. Otterly saw 84 of 62,100 requests over 90 days (0.1%). Evil Martians saw about 770 fetches, of which only 37 came from named AI assistants, across 268k agent requests. Dries Buytaert saw 52 requests in a month, and ‘every one came from SEO audit tools’.

    Wislr watched 48 days and 19 bots, and recorded ‘zero hits from GPTBot, ClaudeBot, PerplexityBot, or any other AI crawler.’ MaxAEO’s 2,400-domain matched-pair study found citation rates of 11.8% with the file against 11.6% without — +0.2pp, inside the noise band.

    llms.txt content quality conventions

    This is the best-evidenced llms.txt signal because it has an exact, documented, shipping consumer whose rules were read in source rather than inferred. Lighthouse core/audits/agentic/llms-txt.js (Copyright 2026 Google LLC) applies three precise tests: hasH1 = /^\s*#\s+.+/m, hasLink = /[.+](.+)/, and isTooShort = content.length < 50. It scores 1 only if all three pass. The failure strings are ‘File is missing a required H1 header’, ‘File does not appear to contain any links’ and ‘File is suspiciously short’.

    This is verifiable and reproducible, independently corroborated by DebugBear and by an SEJ case where a file with working bare URLs failed until links were wrapped in markdown syntax. The spec backs (a): the H1 is ‘the only required section’. Auditing conformance is therefore defensible even though the underlying file’s value to agents is not — this signal grades the parseability claim, which is true, not the retrieval claim, which is refuted.

    Soft-404

    Documented vendor behaviour, and additionally verifiable by construction. Google names the exact failure mode: ‘When a SPA is using client-side JavaScript to handle errors they often report a 200 HTTP status code instead of the appropriate status code. This can lead to error pages being indexed and possibly shown in search results’, with the prescribed fixes being a redirect to a URL that genuinely returns 404, or a robots noindex.

    Google’s status-code reference defines a soft 404 as content that ‘suggests an error… an empty page or an error message’ returned with a 2xx code. The crawl-budget guide states flatly that ‘Soft 404 pages will continue to be crawled, and waste your budget.’ Vercel’s measurements show AI crawlers are far more exposed to that waste than Googlebot: 34.82% of ChatGPT fetches and 34.16% of Claude fetches land on 404s, against 8.22% for Googlebot.

    For the audit tool itself the mechanism is not probabilistic at all — a 200-for-everything origin defeats status-based existence probes deterministically, so soft-404 detection must run as a precondition gate before any other file-presence audit is trusted.

    Limits

    /llms.txt existence at domain root — Google Search Central explicitly names the file: ‘LLMS.txt files and other special markup: You don’t need to create new machine readable files… Doing so will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.’ John Mueller: ‘it’s purely speculative for now (the file has existed for years, yet none of the AI systems use it).’ Gary Illyes: Google does not support it and has no plans to.

    Perplexity’s crawler doc names only robots.txt. OpenAI’s crawler doc never mentions it. No W3C or IETF standing — a search of w3.org found only unrelated AI groups. The widely-repeated ‘June 2026 W3C proposal to standardize llms.txt’ appears to be SEO blogspam with no primary source. Crucially, Lighthouse itself returns notApplicable (score 1) on HTTP 404 — even Google does not penalize absence. Beware the vendor-incentive trap: the main pro-adoption data comes from Mintlify and Profound, who sell llms.txt tooling, with no published methodology.

    llms.txt content quality conventions — Sharply scope this signal. The blockquote convention is optional in the spec, unchecked by Lighthouse, and not universally followed even by exemplars. Across 7 major vendor files measured for this dossier, all 7 had an H1 and markdown links, but only 5 had a blockquote. Anthropic and Stripe both omit it, so failing sites for a missing blockquote would fail Anthropic’s own file. Lighthouse checks no link validity, no per-link descriptions, and no section structure.

    SEJ’s caveat is the honest frame: ‘The audit checks whether your file is mechanically parseable. It does not check whether the file describes your website usefully.’ And parseability is worth little when the consumer population is near-empty — Lighthouse is an auditor, not an agent, and it scores a 404 as notApplicable. Link-validity checking is also expensive at scale (Vercel’s file has 1,872 links).

    Soft-404 — No AI vendor publishes its own soft-404 heuristic. The specific detection thresholds are therefore Google-derived, and the claim that GPTBot or ClaudeBot penalise soft 404s is not directly documented. What is documented is that they waste a third of their fetches on error responses. Detection also has a false-positive risk of its own: a legitimately-configured site may return 200 for a probe path that happens to exist, and some CDNs return 200 with a custom error body by design. The audit should therefore verify via body content (shell fingerprint, absence of expected markers) and not by status code alone, and should report soft-404 as a confidence-degrading condition rather than a page-quality failure.

    Sources