Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    content-extraction/server-responsiveness

    Server responsiveness

    What it checks

    One TTFB audit: the median across every page the crawl actually measured, graded in bands.

    State Result
    the scan was blocked by a WAF, or no page fetch completed (including an empty crawl) na
    median TTFB ≤ 800ms pass
    median TTFB 801–2500ms warn, priority medium
    median TTFB > 2500ms fail, priority high

    Pages whose fetch errored are excluded from the sample rather than charged the timeout value, and the found string states that the figure includes DNS, TCP and TLS setup measured from the scanner’s location.

    Why it matters

    Higher time-to-first-byte reduces the number of URLs an AI crawler fetches per unit time. It also raises the probability that a user-triggered agent fetch is abandoned before bytes arrive, which the origin logs as HTTP 499, client-closed-request. Both reduce the page’s eligibility to be cited.

    Evidence

    TTFB

    Documented first-party for the Google side: ‘If the site slows down (latency increases or response times become longer)… the limit goes down and Google crawls less’, with the crawl capacity limit explicitly described as hostload. Google also documents that 5xx and 429 responses ‘prompt Google’s crawlers to temporarily slow down with crawling’, proportional to the share of erroring URLs — which matters for AI because AI Overviews/AI Mode eligibility flows through that same index.

    Anthropic corroborates load-sensitivity indirectly by being the one AI vendor that explicitly supports robots.txt Crawl-delay. On the AI-specific side the evidence is empirical, but second-party. Profound’s April 2026 analysis of a random 700K-page sample found that pages with fetch-failure rates above 75% received roughly 18x fewer citation events, many of them zero. Log-based reporting attributes clusters of HTTP 499s to OAI-SearchBot, ChatGPT-User and GPTBot on slow origins.

    Limits

    No AI vendor publishes a timeout threshold — not OpenAI, not Anthropic, not Perplexity — so any specific number (the widely repeated ‘1–5 second’ budget) is unsourced folklore and must not be cited as vendor guidance. The 499 research is reported second-hand with no published sample size or methodology, and the Profound figure is a correlation between failure rate and citation count that is plausibly confounded by site quality and authority. Note also the direction is not purely ‘faster is better for the publisher’: Vercel’s data shows GPTBot and ClaudeBot generating 569M and 370M requests/month respectively on one network, so a faster origin also invites more uncompensated crawl.

    How it scores

    First-party for the Google side and stated causally: “If the site slows down (latency increases or response times become longer)… the limit goes down and Google crawls less”, with the crawl capacity limit described as hostload. That is a documented consumer, but for one crawler and with no threshold. No AI vendor publishes a timeout, so the widely repeated “1–5 second” budget is folklore and the audit never cites it as vendor guidance. The abandonment research is second-hand with no published sample size. The audit therefore scores response time as a crawl-budget signal, which is what the evidence supports, and not as a citation-rate signal.

    Sources