Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/duckassistbot

    DuckAssistBot allowed

    What it checks

    Reads the robots.txt rules that apply to DuckAssistBot — its own group if it has one, otherwise the catch-all — and reports whether they let it fetch the site root. A named group is not required: under RFC 9309 §2.2.1 an open catch-all grants the same access.

    Why it matters

    Disallowing DuckAssistBot removes the site as a real-time source for DuckDuckGo’s AI-assisted answers (effective after ~72 hours) without affecting organic DuckDuckGo search rankings; the crawl is documented as never used for model training.

    Evidence

    DuckAssistBot allow/block state in robots.txt

    DuckDuckGo publishes an unusually precise help page. It names the token, ‘DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)’, and the purpose: ‘DuckAssistBot is a web crawler for DuckDuckGo Search that crawls pages in real-time for our AI-assisted answers’. It rules out training use: ‘This data is not used in any way to train AI models’. It states the latency of an opt-out. A disallow ‘will take effect after 72 hours and DuckAssistBot will stop crawling your site’. It also gives a decoupling guarantee: ‘Opting out of DuckAssistBot does not impact organic search rankings.’ All three audit-relevant facts — compliance, latency and non-training use — are vendor-stated and falsifiable.

    Limits

    DuckAssistBot does not appear in Cloudflare Radar’s named AI-crawler top-five breakdowns, so its traffic volume — and therefore the practical stakes of allowing or blocking it — is small relative to GPTBot/ClaudeBot/ChatGPT-User. The 72-hour enforcement lag means a disallow is not immediate.

    How it scores

    DuckDuckGo’s help page is unusually precise. It publishes the token, and states that the crawler “crawls pages in real-time for our AI-assisted answers” and that “This data is not used in any way to train AI models”. It even names the enforcement lag: a disallow takes effect after roughly 72 hours. A vendor documenting the token, the use and the timing is well past the grade-A bar. The practical stakes are small, since the bot does not appear in Cloudflare Radar’s named top-five breakdowns, but that is a question of volume rather than of evidence.

    Sources