Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/amazonbot

    Amazonbot allowed

    What it checks

    Without an explicit robots.txt rule, Amazonbot may still crawl your site but has no signal that it is welcome. Adding an explicit allow rule improves your visibility in AI-powered search and ensures consistent crawler behavior.

    Why it matters

    Disallowing Amazonbot stops Amazon crawling the site for product/service improvement and possible Amazon AI model training; Amazon states the bot honors user-agent and allow/disallow directives but ignores crawl-delay.

    Evidence

    Amazonbot allow/block state in robots.txt

    Amazon documents the UA as ‘Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36’. It states the purpose — ‘Amazonbot is used to improve our products and services’ — and that the data ‘may be used to train Amazon AI models’. It states compliance too: ‘Automated crawling from these listed user agents respects the Robots Exclusion Protocol, honoring the user-agent and the allow/disallow directives.’ The bot is active at scale: Cloudflare Radar ranked Amazonbot second only to GPTBot in the Computer & Electronics vertical (Aug 2025). Amazon also now documents a separate Amzn-SearchBot which ‘does not crawl content for generative AI model training’, so audits should treat the two tokens distinctly.

    Limits

    Explicit vendor negative on a related directive: ‘They do not support the crawl-delay directive’ — so any audit that recommends crawl-delay for Amazonbot is recommending a no-op. Amazon publishes no consequence statement for blocking (no equivalent of OpenAI’s search-exclusion warning), so the visibility cost of a block is undocumented.

    How it scores

    Amazon publishes a crawler page naming the exact product token. It states the purpose: “Amazonbot is used to improve our products and services”, and content “may be used to train Amazon AI models”. It asserts robots.txt compliance for that user agent. That is documented consumer behaviour for the directive this audit reads, which is the grade-A bar. What Amazon does not publish is any statement of what blocking costs, so the audit reports the directive’s state and does not argue a visibility case.

    Sources