Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    machine-discovery/no-broken-links

    No broken internal links

    What it checks

    Broken internal links create dead ends for AI crawlers and waste their limited crawl budget.

    Why it matters

    Soft-404 — A server that returns HTTP 200 with an application shell for URLs that do not exist causes two distinct harms. Crawlers spend capacity on valueless error pages, and may index them. And any automated audit that infers file existence from a 2xx status produces false positives — reporting llms.txt, robots.txt, feeds or .md mirrors as ‘present’ when the origin merely echoed the SPA shell.

    Broken links — Internal links and sitemap entries pointing at 4xx/5xx URLs consume AI crawl budget on dead fetches and cause task-executing agents to abandon navigation, so reducing them raises the ratio of successful content fetches per crawl. Falsifiable: if a site’s AI-crawler 404 rate and citation coverage are unaffected by fixing internal broken links, the claim fails.

    Evidence

    Soft-404

    Documented vendor behaviour, and additionally verifiable by construction. Google names the exact failure mode: ‘When a SPA is using client-side JavaScript to handle errors they often report a 200 HTTP status code instead of the appropriate status code. This can lead to error pages being indexed and possibly shown in search results’, with the prescribed fixes being a redirect to a URL that genuinely returns 404, or a robots noindex.

    Google’s status-code reference defines a soft 404 as content that ‘suggests an error… an empty page or an error message’ returned with a 2xx code. The crawl-budget guide states flatly that ‘Soft 404 pages will continue to be crawled, and waste your budget.’ Vercel’s measurements show AI crawlers are far more exposed to that waste than Googlebot: 34.82% of ChatGPT fetches and 34.16% of Claude fetches land on 404s, against 8.22% for Googlebot.

    For the audit tool itself the mechanism is not probabilistic at all — a 200-for-everything origin defeats status-based existence probes deterministically, so soft-404 detection must run as a precondition gate before any other file-presence audit is trusted.

    This is the best-measured signal in the domain. Vercel and MERJ found ChatGPT spends 34.82% of its fetches on 404 pages and 14.36% following redirects, and Claude 34.16% on 404s — against Googlebot’s 8.22% and 1.49%. Roughly a third of all LLM-crawler effort on a site is landing on nothing. Apple documents the consequence as vendor policy: “Applebot’s crawl rate adjusts automatically when a site slows down or returns errors.” It adds that ‘Identifying content that doesn’t need to be crawled lowers infrastructure costs for site owners’.

    Error rate directly throttles how much of your site Apple fetches. Oncrawl’s production logs show the acute form: 988 ChatGPT-User requests returning 404 concentrated over a few hours on a single retailer. Cloudflare’s crawl-to-refer data establishes why the waste matters economically — AI platforms already fetch orders of magnitude more HTML than they refer back, so every dead fetch is pure cost. On the agent side, BrowserArena’s live-web evaluation identifies direct URL navigation as one of three consistent failure categories, concluding that current web agents are brittle on real-world navigation.

    Limits

    Soft-404 — No AI vendor publishes its own soft-404 heuristic. The specific detection thresholds are therefore Google-derived, and the claim that GPTBot or ClaudeBot penalise soft 404s is not directly documented. What is documented is that they waste a third of their fetches on error responses. Detection also has a false-positive risk of its own: a legitimately-configured site may return 200 for a probe path that happens to exist, and some CDNs return 200 with a custom error body by design. The audit should therefore verify via body content (shell fingerprint, absence of expected markers) and not by status code alone, and should report soft-404 as a confidence-degrading condition rather than a page-quality failure.

    Broken links — An attribution caveat must be published with the claim. The Vercel 404 rates measure URLs the crawlers requested. Oncrawl shows that a large share of those are model-hallucinated or stale-memory paths which the site never linked — ‘ChatGPT inventing URLs’. So the headline 34.82% figure does not isolate the effect of on-site broken links, and fixing every internal 404 would not bring it near Googlebot’s 8.22%. No study demonstrates that repairing internal broken links raises AI citation rate.

    Google’s nofollow guidance separately notes pages ‘may be found through other means’, reinforcing that the crawler’s URL set is not solely derived from your link graph. The correct scored claim is crawl-efficiency and agent-navigation reliability, not citation lift; serving a proper 404/410 and maintaining redirects is the actionable core.

    How it scores

    Soft-404 — Vendor-documented and verifiable by construction. Google names the failure mode exactly: “When a SPA is using client-side JavaScript to handle errors they often report a 200 HTTP status code instead of the appropriate status code. This can lead to error pages being indexed and possibly shown in search results.” The second harm needs no citation at all. A checker that infers a file exists from a 2xx status reports llms.txt or a feed as present on a site that serves an application shell for every path — a defect in the measurement itself. What is not documented is any AI vendor’s own soft-404 heuristic, so the audit claims wasted fetches, not a ranking penalty.

    Broken links — The best-measured signal in the domain: Vercel and MERJ found ChatGPT spends 34.82% of its fetches on 404 pages and Claude 34.16%, against Googlebot’s 8.22%. Roughly a third of all LLM-crawler effort on a site lands on nothing. It stays at B rather than A because of an attribution problem the audit states openly: those figures count URLs the crawlers requested, and Oncrawl shows a large share are model-hallucinated or stale-memory paths the site never linked. So the headline number does not isolate the effect of on-site broken links, and the audit claims wasted crawl budget rather than a measured citation loss.

    Sources