Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/crawl-delay

    Crawl-delay is reasonable

    What it checks

    Excessive Crawl-delay values (over 10 seconds) dramatically slow AI indexing, meaning your latest content may take days or weeks to appear in AI search results.

    Why it matters

    A Crawl-delay line throttles fetch rate only for the specific crawlers that implement it. It is not part of RFC 9309, and the largest consumers explicitly do not support it. Its presence cannot be scored as a general readiness improvement.

    Evidence

    Crawl-delay directive in robots.txt

    Support is genuinely split and each side is vendor-documented. Supporting: Anthropic explicitly names support for the ‘Crawl-delay extension to robots.txt’ for its bots; You.com states ‘YouBot fully respects robots.txt directives, including user-agent specific rules and crawl-delay settings’. Not supporting: Google states ‘Google supports the following fields (other fields such as crawl-delay aren’t supported)’ — only user-agent, allow, disallow and sitemap; Amazon states of Amazonbot ‘They do not support the crawl-delay directive.’

    Limits

    RFC 9309 defines no crawl-delay directive at all — the only accommodation is §2.2.4, which merely permits crawlers to ‘interpret other records that are not part of the robots.txt protocol’. There is no interoperable value semantics (seconds vs. requests-per-second is unspecified), no vendor consensus, and the two highest-volume AI crawlers in Cloudflare’s data (GPTBot, ClaudeBot’s operator aside) publish no crawl-delay commitment. OpenAI, Perplexity, Meta, Mistral and DuckDuckGo document no crawl-delay support in either direction. A crawl-delay line is at best a per-vendor hint; never score its presence or absence.

    Sources