Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    machine-discovery/llms-txt-link-descriptions

    llms.txt links include descriptions

    What it checks

    How many markdown links in /llms.txt carry a description after the URL, in the form - [Name](url): description. The format makes descriptions optional.

    Why it matters

    No AI vendor documents reading llms.txt or llms-full.txt from a website at runtime. An Ahrefs study of 137,210 domains (verified 2026-10-18) found that 97% of llms.txt files received no requests in a month. AI tools never asked for a missing file. Tools a user points at the file, such as documentation helpers in coding agents, can still read it. Link descriptions are optional in the format.

    Evidence

    Mechanism claim: An agent reading /llms.txt uses the : description note after each link to decide which of the linked pages to fetch, so links without a note cause the agent to fetch pages it would otherwise skip.

    Grade: C — the note is a spec-defined optional element, and the format’s reference parser exposes it as a per-link desc field. Vendor tooling, Chrome Lighthouse included, treats the link list itself as the point of the file. But no vendor documents a named agent pruning fetches on the basis of those notes.

    • The llms.txt spec defines the link entry as “a required markdown hyperlink [name](url), then optionally a : and notes about the file” — the description is part of the format, and explicitly optional — llmstxt.org (verified 2026-08-21)
    • The reference implementation emits each link as an object with title, url and an optional desc, so the note is a first-class parsed field rather than free text — raw.githubusercontent.com/…/core.py (verified 2026-08-21)
    • The llms_txt2ctx CLI built on that parser expands the listed links into a single LLM context document, which is the concrete consumption path the audit’s rationale assumes — pypi.org/…/llms-txt (verified 2026-08-21)
    • Chrome’s agentic-browsing Lighthouse audit fails an llms.txt that contains no markdown links at all. The test is /\[.+\]\(.+\)/, and the message is “File does not appear to contain any links”. That corroborates that the described link list, not the prose, is the payload — github.com/…/llms-txt.js (verified 2026-08-21)

    Limits

    The same Lighthouse audit stops at “contains at least one link” and checks nothing about descriptions, so even the one vendor-shipped llms.txt checker does not treat the note as required (github.com/…/llms-txt.js, verified 2026-08-21). The spec marks the note “optional” (llmstxt.org, verified 2026-08-21), so the audit’s 50%-described FAIL threshold has no basis in any published source. Google states Search ignores llms.txt and that no AI text file is needed for its generative features (developers.google.com/…/ai-optimization-guide, verified 2026-08-21).

    Measured traffic shows named AI assistants accounted for only 37 of ~770 llms.txt fetches over two months, while ~15% of agent page reads came through Accept:-negotiated Markdown instead — i.e. the observed agent path bypasses llms.txt (evilmartians.com/…/which-ai-actually-reads-your-sit…, verified 2026-08-21). OpenAI (developers.openai.com/…/bots) and Perplexity (docs.perplexity.ai/…/perplexity-crawlers) mention llms.txt on their crawler pages only as the index of their own documentation. Neither says its crawlers read another site’s llms.txt (both verified 2026-10-18).

    How it scores

    Informative, weight 0. Every result carries an unconsumed notice: no AI vendor documents reading this file. The audit never warns or fails. An absent file is not applicable. A file that is present passes, and links without descriptions, or a file with no links, are named in the pass message, with the note that it does not affect AI agents.

    Sources