Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/crawl-delay

    Crawl-delay is reasonable

    What it checks

    The Crawl-delay values a site’s robots.txt sets, and whether any is above 10 seconds. Only the crawlers that implement the directive slow down for it. Anthropic, Common Crawl and You.com document support for it. Google, Amazon and Apple document that they ignore it, so a group addressed only to one of their crawlers is not judged.

    State Result
    no robots.txt na
    robots.txt sets no Crawl-delay na
    Crawl-delay set only for Google, Amazonbot or Applebot na
    every other Crawl-delay is 10 s or less pass
    any other Crawl-delay is above 10 s fail, priority high

    The 10-second line follows Bing: “We recommend against using any value higher than 10” (blogs.bing.com/…/crawl-delay-and-the-bing-crawler-m…, verified 2026-10-18). No vendor documents how long a large delay holds back inclusion in AI search.

    The audit is informative and carries weight 0, so no verdict changes a score. The directive is not part of RFC 9309, so its absence is not a defect.

    Why it matters

    A Crawl-delay line throttles fetch rate only for the specific crawlers that implement it. It is not part of RFC 9309, and the largest consumers explicitly do not support it. Its presence cannot be scored as a general readiness improvement.

    Evidence

    Crawl-delay directive in robots.txt

    Support is genuinely split and each side is vendor-documented. Supporting: Anthropic explicitly names support for ‘the non-standard Crawl-delay extension to robots.txt’ for its bots (support.claude.com/…/8896518-does-anthropic-crawl-d…, verified 2026-10-17); You.com states ‘YouBot fully respects robots.txt directives, including user-agent specific rules and crawl-delay settings’; Common Crawl states ‘We obey the Crawl-delay parameter for robots.txt’ (commoncrawl.org/faq, verified 2026-10-18). Not supporting: Google states ‘Google supports the following fields (other fields such as crawl-delay aren’t supported)’ — only user-agent, allow, disallow and sitemap (developers.google.com/…/robots_txt, verified 2026-10-17); Amazon states of Amazonbot ‘They do not support the crawl-delay directive.’; Apple states ‘Applebot does not follow crawl-delay.’ (support.apple.com/…/119829, verified 2026-10-18)

    Limits

    RFC 9309 defines no crawl-delay directive at all — the only accommodation is §2.2.4, which merely permits crawlers to ‘interpret other records that are not part of the robots.txt protocol’. There is no interoperable value semantics (seconds vs. requests-per-second is unspecified), no vendor consensus, and the two highest-volume AI crawlers in Cloudflare’s data (GPTBot, ClaudeBot’s operator aside) publish no crawl-delay commitment. OpenAI, Perplexity, Meta, Mistral and DuckDuckGo document no crawl-delay support in either direction. A crawl-delay line is at best a per-vendor hint; never score its presence or absence.

    Sources