Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    machine-discovery/no-broken-ai-endpoints

    No broken AI endpoints

    What it checks

    AI agents follow URLs in your ai-catalog.json, llms.txt, and navigation.json to build a map of your site’s AI-consumable resources. Broken links cause agents to lose trust in your manifest files entirely, potentially ignoring all listed endpoints. Fix or remove broken URLs.

    Why it matters

    Soft-404 — A server that returns HTTP 200 with an application shell for URLs that do not exist causes two distinct harms. Crawlers spend capacity on valueless error pages, and may index them. And any automated audit that infers file existence from a 2xx status produces false positives — reporting llms.txt, robots.txt, feeds or .md mirrors as ‘present’ when the origin merely echoed the SPA shell.

    WAF — Edge bot-management sitting in front of the origin can deny AI crawlers and agents outright, or gate them behind payment, so a site with perfect content structure can still be entirely invisible to AI systems.

    Evidence

    Soft-404

    Documented vendor behaviour, and additionally verifiable by construction. Google names the exact failure mode: ‘When a SPA is using client-side JavaScript to handle errors they often report a 200 HTTP status code instead of the appropriate status code. This can lead to error pages being indexed and possibly shown in search results’, with the prescribed fixes being a redirect to a URL that genuinely returns 404, or a robots noindex.

    Google’s status-code reference defines a soft 404 as content that ‘suggests an error… an empty page or an error message’ returned with a 2xx code. The crawl-budget guide states flatly that ‘Soft 404 pages will continue to be crawled, and waste your budget.’ Vercel’s measurements show AI crawlers are far more exposed to that waste than Googlebot: 34.82% of ChatGPT fetches and 34.16% of Claude fetches land on 404s, against 8.22% for Googlebot.

    For the audit tool itself the mechanism is not probabilistic at all — a 200-for-everything origin defeats status-based existence probes deterministically, so soft-404 detection must run as a precondition gate before any other file-presence audit is trusted.

    WAF

    Fully documented vendor behaviour at large scale. Cloudflare began blocking AI crawlers by default for new domains on 1 July 2025, making permission the default posture across a very large share of the web. It has since announced a change for 15 September 2026. New domains will block ‘Training’ and ‘Agent’ class bots by default on ad-displaying pages, while leaving ‘Search’ allowed, and has extended managed robots.txt with a use signal (immediate / reference / full).

    Pay-per-crawl operationalises the gate in HTTP. Crawlers ‘either present payment intent via request headers for successful HTTP 200 access, or receive an HTTP 402 Payment Required response with pricing’, at a minimum of $0.001 per crawl. Critically, an existing WAF or Bot Management block rule overrides pay-per-crawl’s charge behaviour, and silently converts a monetizable crawl into a hard block. Adoption context: of 3,816 top-10k domains with robots.txt, ~14% carried AI-bot directives; GPTBot was the most-disallowed at 312 domains.

    Verification is moving to cryptography — Cloudflare’s Web Bot Auth implements RFC 9421 HTTP Message Signatures with Signature-Input / Signature / Signature-Agent headers and a JWKS key directory at /.well-known/http-message-signatures-directory.

    Limits

    Soft-404 — No AI vendor publishes its own soft-404 heuristic. The specific detection thresholds are therefore Google-derived, and the claim that GPTBot or ClaudeBot penalise soft 404s is not directly documented. What is documented is that they waste a third of their fetches on error responses. Detection also has a false-positive risk of its own: a legitimately-configured site may return 200 for a probe path that happens to exist, and some CDNs return 200 with a custom error body by design. The audit should therefore verify via body content (shell fingerprint, absence of expected markers) and not by status code alone, and should report soft-404 as a confidence-degrading condition rather than a page-quality failure.

    WAF — Two caveats that shape how this must be audited. (1) User-Agent-based probing is unreliable in both directions: Cloudflare documented Perplexity using a Chrome-impersonating stealth crawler on unlisted IPs across rotating ASNs at 3–6M requests/day to evade no-crawl directives, and conversely malicious scrapers routinely spoof GPTBot. An audit that merely sets a UA header measures the WAF’s UA rules, not real agent access — genuine verification requires the vendors’ published IP ranges (openai.com/gptbot.json, claude.com/crawling/bots.json, perplexity.com/perplexitybot.json) or Web Bot Auth signatures. (2) Blocking is often a deliberate, rational business decision, not a defect.

    Cloudflare’s crawl-to-refer data puts Anthropic at about 71,000 crawls per HTML referral in the June 2025 window, with the caveat that Claude’s native app sends no Referer. That makes uncompensated crawl a real cost. The audit should REPORT the gate neutrally as ‘AI agents are blocked here’ rather than scoring it as a failure, since the site owner may have chosen it.

    How it scores

    Soft-404 — Vendor-documented and verifiable by construction. Google names the failure mode exactly: “When a SPA is using client-side JavaScript to handle errors they often report a 200 HTTP status code instead of the appropriate status code. This can lead to error pages being indexed and possibly shown in search results.” The second harm needs no citation at all. A checker that infers a file exists from a 2xx status reports llms.txt or a feed as present on a site that serves an application shell for every path — a defect in the measurement itself. What is not documented is any AI vendor’s own soft-404 heuristic, so the audit claims wasted fetches, not a ranking penalty.

    WAF — Vendor-documented at a scale that changes the default posture of the web. Cloudflare began blocking AI crawlers by default for new domains on 1 July 2025. It has announced that from 15 September 2026, new domains on ad-displaying sites block Training and Agent class bots by default. A named intermediary publishing what it blocks, and when, is well past the grade-A bar — a site with perfect structure can be entirely invisible behind it. How it is audited is deliberately narrow, because User-Agent probing is unreliable in both directions: Cloudflare documented Perplexity impersonating Chrome from unlisted IPs at 3–6M requests a day, and scrapers routinely spoof GPTBot.

    Sources