Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/claude-user

    Claude-User allowed

    What it checks

    Without an explicit robots.txt rule, Claude-User may still crawl your site but has no signal that it is welcome. Adding an explicit allow rule improves your visibility in AI-powered search and ensures consistent crawler behavior.

    Why it matters

    Disallowing Claude-User prevents Claude from fetching the site when a user asks Claude about it. Unlike OpenAI, Perplexity and Meta, Anthropic asserts no user-initiated exemption — its blanket robots.txt-compliance statement covers Claude-User, so the directive is expected to be honored.

    Evidence

    Claude-User allow/block state in robots.txt

    Anthropic documents Claude-User as: ‘supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent’, under the same statement that ‘Anthropic’s Bots respect do not crawl signals by honoring industry standard directives in robots.txt’. This makes Claude-User a rare case where blocking a user-initiated agent is documented to actually work — which also means blocking it is a real self-inflicted visibility cost. TollBit measured real-world blocking at 9% of European sites vs 26% of North American sites, confirming active field presence and operator awareness.

    Limits

    Anthropic does not publish a separate per-agent compliance statement, only a blanket one, so the Claude-User guarantee is weaker than a dedicated sentence. No independent audit has specifically measured Claude-User disallow compliance the way TollBit did for ChatGPT-User, so the honoring claim is vendor-asserted and untested.

    How it scores

    Anthropic documents the token — “When individuals ask questions to Claude, it may access websites using a Claude-User agent” — under a compliance statement that covers it without carving out a user-initiated exemption. That matters: OpenAI, Perplexity, Meta and Mistral all reserve one for their equivalent agents, so this is a rare case where a directive aimed at a user-triggered fetcher is expected to be honoured. Named agent, named directive, stated behaviour: grade A. The guarantee rests on a blanket statement rather than a per-agent one, and no independent audit has tested it, so the audit reports the state rather than promising enforcement.

    Sources