Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    machine-discovery/root-text-file-resolution-integrity

    The origin serves and correctly 404s root-level .txt resources

    What it checks

    Proves the origin can actually serve and correctly 404 root-level .txt resources — the physical precondition for IndexNow key verification and for every other .txt-based agent discovery surface (llms.txt, ai.txt, security.txt, ads.txt). Sites behind SPA rewrites, WAF challenge pages, or HTML-404-with-200 handlers silently fail all of them.

    Why it matters

    IndexNow proves ownership by fetching host/%7Bkey%7D.txt and byte-comparing the body to {key}; a non-matching body yields HTTP 403 (‘key not found in file’) and the submission is discarded by every participating engine (Bing, Yandex, Naver, Seznam, Yep, Amazon). Falsifiable claim: if GET host<random-32-hex>.txt returns 200 rather than 404, the origin has a catch-all that returns non-key content for arbitrary root .txt paths. Key rotation, key removal and key-file health are then undetectable. The same catch-all makes every probe-based discovery file — llms.txt, ai.txt, security.txt — indistinguishable from a soft-404. Predicts: sites failing this probe cannot be given a trustworthy ‘llms.txt present’ verdict either, because a 200 response there carries no information.

    Evidence

    • IndexNow Protocol Documentation — IndexNow (Microsoft/Yandex) (spec, URL verified 2026-08-20)
    • Ownership is proven by hosting a UTF-8 text file at the host root named {key}.txt whose body is the key. Key must be 8-128 chars from [a-zA-Z0-9-]. Verification is a byte comparison. HTTP 403 is returned when the key is ‘not found in the key file’ or invalid. 422 signals a host or schema mismatch, 429 a rate limit, and 202 means ‘key validation pending’. keyLocation restricts submittable URLs to the key file’s directory and deeper. Batch POST accepts up to 10,000 URLs.
    • IndexNow FAQ — participating search engines — IndexNow (vendor-doc, URL verified 2026-08-20)
    • Participating engines: Amazon, Bing, Naver, Seznam.cz, Yandex, Yep. Submissions to the global endpoint are shared with all participants. States IndexNow ‘helps keep your content current in AI-powered search results’ but names no specific LLM/Copilot consumer — the AI-consumer link is a vendor claim, not a documented pipeline.
    • IndexNow: Instantly Index your Web Content in Search Engines — Microsoft Bing Webmaster Blog (vendor-doc, URL verified 2026-08-20)
    • Confirms the key-file-at-root verification flow and the motivation (organic discovery ‘can take days or even weeks’). No published crawl-latency SLA.

    How it scores

    Tier per evidence policy: scored — grade B meets the A/B bar required for scored audits.

    Example failure

    A Next.js site deployed on a host with a catch-all rewrite /(.*) -> /index.html returns 200 text/html for /a3f2c9d1e4b70856.txt. The owner’s real IndexNow key file also 200s, so Bing Webmaster shows ‘key verified’. Then a framework upgrade reorders the rewrite ahead of static file serving. Every key fetch now returns the app shell, IndexNow starts answering 403 to all submissions, and the site loses push indexing across six engines with no visible symptom. The same origin returns 200 for /llms.txt, so an ‘llms.txt present’ badge from any auditor is a false positive.

    Sources