Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/claude-searchbot

    Claude-SearchBot allowed

    What it checks

    Reads the robots.txt rules that apply to Claude-SearchBot — its own group if it has one, otherwise the catch-all — and reports whether they let it fetch the site root. A named group is not required: under RFC 9309 §2.2.1 an open catch-all grants the same access.

    Why it matters

    Disallowing Claude-SearchBot removes the site from the index Anthropic uses to improve Claude’s search result quality, reducing the chance of being surfaced/cited in Claude search answers.

    Evidence

    Claude-SearchBot allow/block state in robots.txt

    Documented by Anthropic: Claude-SearchBot ‘navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses’, covered by the same robots.txt-compliance statement and the claude.com/crawling/bots.json IP list. As a search/citation-side agent it is the Anthropic analogue of OAI-SearchBot and PerplexityBot, so an allow is the visibility-positive state.

    Limits

    Anthropic does not state the consequence of blocking as explicitly as OpenAI does for OAI-SearchBot (‘will not be shown in ChatGPT search answers’) — the exclusion-from-answers link is inferred from the agent’s stated purpose, not asserted. No published per-bot citation-impact study isolates Claude-SearchBot.

    How it scores

    Anthropic names the token and its purpose — it “analyzes online content specifically to enhance the relevance and accuracy of search responses” — under the same robots.txt-compliance statement and IP list that cover its other bots. That is a named agent reading a named directive, which is grade A. The exclusion-from-answers consequence is inferred from the stated purpose rather than asserted the way OpenAI asserts it for OAI-SearchBot, which is why the audit’s copy states the purpose and not a promise.

    Sources