Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    machine-discovery/ai-file-delivery

    AI files are delivered correctly

    What it checks

    How the AI files the scan already fetched are delivered — two headers per file, one audit:

    File Expected Content-Type
    /llms.txt text/plain or text/markdown
    /.well-known/ai-catalog.json application/json
    /openapi.json application/json (YAML types tolerated)
    /sitemap.xml application/xml or text/xml

    A file counts as cacheable when it carries a Cache-Control with a non-zero max-age and no no-store/no-cache, or an ETag / Last-Modified validator.

    State Result
    every served file correctly typed and cacheable pass
    any file mis-typed fail, priority medium
    types correct, some file with no caching headers warn, priority low
    no AI file was served (or the response is the site’s HTML shell) na

    At tier: informative / weight 0 none of these outcomes moves a score.

    Why it matters

    The claim under test: serving /llms.txt and .md mirrors as text/plain or text/markdown (rather than text/html, application/octet-stream or a wrong charset) is required for AI consumers to parse them correctly.

    Evidence

    Correct Content-Type for llms.txt and .md files

    Convention with sensible precedent, not a documented requirement. RFC 9116 does establish the pattern for well-known plain-text files — security.txt ‘must be served as plain text (MIME type text/plain) with UTF-8 encoding’. The llms.txt spec uses type="text/markdown" when describing link relations, so text/markdown is the intent-consistent choice. Two real failure modes are mechanically certain rather than speculative: application/octet-stream triggers download-rather-than-parse behaviour in browser-based consumers, and a Content-Type of text/html on a Markdown file will lead HTML-oriented extraction pipelines to run an HTML parser over Markdown. X-Content-Type-Options: nosniff, where present, removes the browser’s ability to recover from a wrong type.

    Limits

    The llmstxt.org specification states no requirement for the file’s own HTTP Content-Type — it only mentions text/markdown in the context of link relations. No AI vendor documentation (OpenAI, Anthropic, Perplexity, Google, Apple) specifies a Content-Type requirement for any AI-facing file. LLM ingestion pipelines are in practice tolerant text extractors; there is no published case of a named crawler rejecting a correctly-named llms.txt on Content-Type grounds. The widely repeated claim that ‘some crawlers will refuse application/octet-stream’ traces only to SEO blogs, not to any primary source. Grade C: plausible mechanism, partial adoption, unproven effect.

    Sources