Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    machine-discovery/llms-txt-links-valid

    llms.txt links are valid

    What it checks

    Whether the links inside a published /llms.txt resolve to HTTP 200. Up to 20 links are checked, spread across the file. A link that is refused or unanswered (0, 401, 403, 429) counts as unverifiable, not broken.

    Why it matters

    No AI vendor documents reading llms.txt or llms-full.txt from a website at runtime. An Ahrefs study of 137,210 domains (verified 2026-10-18) found that 97% of llms.txt files received no requests in a month. AI tools never asked for a missing file. Tools a user points at the file, such as documentation helpers in coding agents, can still read it.

    Evidence

    /llms.txt existence at domain root

    Mechanism: Publishing a well-formed /llms.txt at the domain root causes major AI crawlers and agents (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended) to fetch it and use it to navigate the site, measurably increasing retrieval coverage or citation rate.

    Refuted as a retrieval mechanism, and supported only as a signal of convention-following. The spec is real and actively maintained: llmstxt.org v2, 2026-08-10, about 2.6k GitHub stars. Adoption is non-trivial too — 8.7% of the Tranco top-1,000, or 15.8% of the 549 reachable, and about 10.13% across a 300k-domain SE Ranking sample. All four major AI labs publish one, with HTTP 200 verified on Anthropic, OpenAI, Perplexity, Cloudflare, Stripe, GitHub and Vercel.

    Google Chrome ships a real llms-txt audit in Lighthouse 13.3’s Agentic Browsing category — the audit and gatherer source were read directly. But no vendor anywhere documents an agent consuming it. Four independent log studies converge on near-zero AI-crawler interest. Otterly saw 84 of 62,100 requests over 90 days (0.1%). Evil Martians saw about 770 fetches, of which only 37 came from named AI assistants, across 268k agent requests. Dries Buytaert saw 52 requests in a month, and ‘every one came from SEO audit tools’.

    Wislr watched 48 days and 19 bots, and recorded ‘zero hits from GPTBot, ClaudeBot, PerplexityBot, or any other AI crawler.’ MaxAEO’s 2,400-domain matched-pair study found citation rates of 11.8% with the file against 11.6% without — +0.2pp, inside the noise band.

    Counter-evidence: Google Search Central explicitly names the file: ‘LLMS.txt files and other special markup: You don’t need to create new machine readable files… Doing so will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.’ John Mueller: ‘it’s purely speculative for now (the file has existed for years, yet none of the AI systems use it).’ Gary Illyes: Google does not support it and has no plans to.

    Perplexity’s crawler doc names only robots.txt. OpenAI’s crawler doc never mentions it. No W3C or IETF standing — a search of w3.org found only unrelated AI groups. The widely-repeated ‘June 2026 W3C proposal to standardize llms.txt’ appears to be SEO blogspam with no primary source. Crucially, Lighthouse itself returns notApplicable (score 1) on HTTP 404 — even Google does not penalize absence. Beware the vendor-incentive trap: the main pro-adoption data comes from Mintlify and Profound, who sell llms.txt tooling, with no published methodology.

    llms.txt content quality conventions

    Mechanism: An llms.txt that conforms to structural conventions is machine-parseable and passes automated validation, whereas a malformed one fails. Split precisely: (a) H1 present, (b) at least one text markdown link, (c) length >= 50 chars are enforced by a real shipping consumer; (d) blockquote summary, (e) per-link descriptions, (f) link validity are spec-optional with no known enforcing consumer.

    This is the best-evidenced llms.txt signal because it has an exact, documented, shipping consumer whose rules were read in source rather than inferred. Lighthouse core/audits/agentic/llms-txt.js (Copyright 2026 Google LLC) applies three precise tests: hasH1 = /^\s*#\s+.+/m, hasLink = /[.+](.+)/, and isTooShort = content.length < 50. It scores 1 only if all three pass. The failure strings are ‘File is missing a required H1 header’, ‘File does not appear to contain any links’ and ‘File is suspiciously short’.

    This is verifiable and reproducible, independently corroborated by DebugBear and by an SEJ case where a file with working bare URLs failed until links were wrapped in markdown syntax. The spec backs (a): the H1 is ‘the only required section’. Auditing conformance is therefore defensible even though the underlying file’s value to agents is not — this signal grades the parseability claim, which is true, not the retrieval claim, which is refuted.

    Counter-evidence: Sharply scope this signal. The blockquote convention is optional in the spec, unchecked by Lighthouse, and not universally followed even by exemplars. Across 7 major vendor files measured for this dossier, all 7 had an H1 and markdown links, but only 5 had a blockquote. Anthropic and Stripe both omit it, so failing sites for a missing blockquote would fail Anthropic’s own file. Lighthouse checks no link validity, no per-link descriptions, and no section structure.

    SEJ’s caveat is the honest frame: ‘The audit checks whether your file is mechanically parseable. It does not check whether the file describes your website usefully.’ And parseability is worth little when the consumer population is near-empty — Lighthouse is an auditor, not an agent, and it scores a 404 as notApplicable. Link-validity checking is also expensive at scale (Vercel’s file has 1,872 links).

    Limits

    The audit checks at most 20 links, taken evenly from the start to the end of the file, with 4 requests in flight at a time. Each link is fetched with redirects followed. A link that answers 401, 403 or 429, or does not answer, is reported as unverifiable, not broken: the answer is about the scanner, not the link. When no checked link can be verified, the result is not applicable.

    How it scores

    Informative, weight 0. Every result carries an unconsumed notice: no AI vendor documents reading this file. The audit never warns or fails. An absent file is not applicable. A file that is present passes, and broken links, or a file with no links, are named in the pass message, with the note that it does not affect AI agents.

    Sources