Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    machine-discovery/llms-txt-links-valid

    llms.txt links are valid

    What it checks

    Valid links in llms.txt ensure AI agents can navigate to your content without encountering dead ends.

    Why it matters

    /llms.txt existence at domain root — Publishing a well-formed /llms.txt at the domain root causes major AI crawlers and agents (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended) to fetch it and use it to navigate the site, measurably increasing retrieval coverage or citation rate.

    llms.txt content quality conventions — An llms.txt that conforms to structural conventions is machine-parseable and passes automated validation, whereas a malformed one fails. Split precisely: (a) H1 present, (b) at least one text markdown link, (c) length >= 50 chars are enforced by a real shipping consumer; (d) blockquote summary, (e) per-link descriptions, (f) link validity are spec-optional with no known enforcing consumer.

    Evidence

    /llms.txt existence at domain root

    Refuted as a retrieval mechanism, and supported only as a signal of convention-following. The spec is real and actively maintained: llmstxt.org v2, 2026-08-10, about 2.6k GitHub stars. Adoption is non-trivial too — 8.7% of the Tranco top-1,000, or 15.8% of the 549 reachable, and about 10.13% across a 300k-domain SE Ranking sample. All four major AI labs publish one, with HTTP 200 verified on Anthropic, OpenAI, Perplexity, Cloudflare, Stripe, GitHub and Vercel.

    Google Chrome ships a real llms-txt audit in Lighthouse 13.3’s Agentic Browsing category — the audit and gatherer source were read directly. But no vendor anywhere documents an agent consuming it. Four independent log studies converge on near-zero AI-crawler interest. Otterly saw 84 of 62,100 requests over 90 days (0.1%). Evil Martians saw about 770 fetches, of which only 37 came from named AI assistants, across 268k agent requests. Dries Buytaert saw 52 requests in a month, and ‘every one came from SEO audit tools’.

    Wislr watched 48 days and 19 bots, and recorded ‘zero hits from GPTBot, ClaudeBot, PerplexityBot, or any other AI crawler.’ MaxAEO’s 2,400-domain matched-pair study found citation rates of 11.8% with the file against 11.6% without — +0.2pp, inside the noise band.

    llms.txt content quality conventions

    This is the best-evidenced llms.txt signal because it has an exact, documented, shipping consumer whose rules were read in source rather than inferred. Lighthouse core/audits/agentic/llms-txt.js (Copyright 2026 Google LLC) applies three precise tests: hasH1 = /^\s*#\s+.+/m, hasLink = /[.+](.+)/, and isTooShort = content.length < 50. It scores 1 only if all three pass. The failure strings are ‘File is missing a required H1 header’, ‘File does not appear to contain any links’ and ‘File is suspiciously short’.

    This is verifiable and reproducible, independently corroborated by DebugBear and by an SEJ case where a file with working bare URLs failed until links were wrapped in markdown syntax. The spec backs (a): the H1 is ‘the only required section’. Auditing conformance is therefore defensible even though the underlying file’s value to agents is not — this signal grades the parseability claim, which is true, not the retrieval claim, which is refuted.

    Limits

    /llms.txt existence at domain root — Google Search Central explicitly names the file: ‘LLMS.txt files and other special markup: You don’t need to create new machine readable files… Doing so will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.’ John Mueller: ‘it’s purely speculative for now (the file has existed for years, yet none of the AI systems use it).’ Gary Illyes: Google does not support it and has no plans to.

    Perplexity’s crawler doc names only robots.txt. OpenAI’s crawler doc never mentions it. No W3C or IETF standing — a search of w3.org found only unrelated AI groups. The widely-repeated ‘June 2026 W3C proposal to standardize llms.txt’ appears to be SEO blogspam with no primary source. Crucially, Lighthouse itself returns notApplicable (score 1) on HTTP 404 — even Google does not penalize absence. Beware the vendor-incentive trap: the main pro-adoption data comes from Mintlify and Profound, who sell llms.txt tooling, with no published methodology.

    llms.txt content quality conventions — Sharply scope this signal. The blockquote convention is optional in the spec, unchecked by Lighthouse, and not universally followed even by exemplars. Across 7 major vendor files measured for this dossier, all 7 had an H1 and markdown links, but only 5 had a blockquote. Anthropic and Stripe both omit it, so failing sites for a missing blockquote would fail Anthropic’s own file. Lighthouse checks no link validity, no per-link descriptions, and no section structure.

    SEJ’s caveat is the honest frame: ‘The audit checks whether your file is mechanically parseable. It does not check whether the file describes your website usefully.’ And parseability is worth little when the consumer population is near-empty — Lighthouse is an auditor, not an agent, and it scores a 404 as notApplicable. Link-validity checking is also expensive at scale (Vercel’s file has 1,872 links).

    Sources