Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    machine-discovery/llms-txt-link-descriptions

    llms.txt links include descriptions

    What it checks

    Link descriptions help AI agents understand what each page covers without visiting it, reducing unnecessary crawling.

    Why it matters

    An agent reading /llms.txt uses the : description note after each link to decide which of the linked pages to fetch, so links without a note cause the agent to fetch pages it would otherwise skip.

    Evidence

    • The llms.txt spec defines the link entry as “a required markdown hyperlink [name](url), then optionally a : and notes about the file” — the description is part of the format, and explicitly optional — llmstxt.org (verified 2026-08-21)
    • The reference implementation emits each link as an object with title, url and an optional desc, so the note is a first-class parsed field rather than free text — raw.githubusercontent.com/…/core.py (verified 2026-08-21)
    • The llms_txt2ctx CLI built on that parser expands the listed links into a single LLM context document, which is the concrete consumption path the audit’s rationale assumes — pypi.org/…/llms-txt (verified 2026-08-21)
    • Chrome’s agentic-browsing Lighthouse audit fails an llms.txt that contains no markdown links at all. The test is /\[.+\]\(.+\)/, and the message is “File does not appear to contain any links”. That corroborates that the described link list, not the prose, is the payload — github.com/…/llms-txt.js (verified 2026-08-21)

    Limits

    The same Lighthouse audit stops at “contains at least one link” and checks nothing about descriptions, so even the one vendor-shipped llms.txt checker does not treat the note as required (github.com/…/llms-txt.js, verified 2026-08-21). The spec marks the note “optional” (llmstxt.org, verified 2026-08-21), so the audit’s 50%-described FAIL threshold has no basis in any published source. Google states Search ignores llms.txt and that no AI text file is needed for its generative features (developers.google.com/…/ai-optimization-guide, verified 2026-08-21).

    Measured traffic shows named AI assistants accounted for only 37 of ~770 llms.txt fetches over two months, while ~15% of agent page reads came through Accept:-negotiated Markdown instead — i.e. the observed agent path bypasses llms.txt (evilmartians.com/…/which-ai-actually-reads-your-sit…, verified 2026-08-21). OpenAI (developers.openai.com/…/bots) and Perplexity (docs.perplexity.ai/…/perplexity-crawlers) document robots.txt only and never mention llms.txt (both verified 2026-08-21).

    How it scores

    The note is a spec-defined optional element, and the format’s reference parser exposes it as a per-link desc field. Vendor tooling, Chrome Lighthouse included, treats the link list itself as the point of the file. But no vendor documents a named agent pruning fetches on the basis of those notes.

    Sources