Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/agent-governance

    AI crawler vs conversational agent separation

    What it checks

    Not all AI bots are the same. Training crawlers like GPTBot, CCBot, and Google-Extended scrape your content to build datasets, while conversational and retrieval agents like ChatGPT-User, Claude-User, and OAI-SearchBot fetch pages live to answer real user questions and can send referral traffic back to you. Many sites want to block the former while welcoming the latter — but a single catch-all User-agent: * cannot express that distinction. Granular robots.txt governance names both categories explicitly so each gets the access policy you actually intend.

    State Result
    no robots.txt na
    no AI agent named, and the catch-all allows access or there is none na — nothing to separate
    no AI agent named, and the catch-all blocks the whole site fail, priority medium
    both categories named, two of each or with different policies pass
    only one category named, or too few of each to differ warn, priority medium

    Why it matters

    Each major AI vendor operates separate robots.txt product tokens for dataset-training crawling and for live retrieval or search grounding. Per RFC 9309, a crawler obeys the group matching its own token, and falls back to * only when no such group exists. A robots.txt that names the two categories separately therefore produces different access outcomes for training and for live retrieval — an outcome a catch-all group alone cannot express.

    Evidence

    • RFC 9309 §2.2.1: “Crawlers MUST use case-insensitive matching to find the group that matches the product token and then obey the rules of the group”. If no specific match exists, “crawlers MUST obey the group with a user-agent line with the ‘*’ value, if present”. A specific group therefore overrides the catch-all for that agent — the exact capability the audit measures — rfc-editor.org/…/rfc9309.html (verified 2026-08-21)
    • OpenAI documents the split directly. GPTBot is “used to make our generative AI foundation models more useful and safe”, and “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models”. OAI-SearchBot is “used to surface websites in search results in ChatGPT’s search features”, and “we recommend allowing OAI-SearchBot in your site’s robots.txt file”. ChatGPT-User is user-initiated — developers.openai.com/…/bots (verified 2026-08-21)
    • Anthropic operates three tokens with distinct purposes — ClaudeBot (training-corpus collection), Claude-User (user-initiated fetches), Claude-SearchBot (search-quality analysis) — and documents per-agent robots.txt groups (User-agent: ClaudeBot / Disallow: /) — support.claude.com/…/8896518-does-anthropic-crawl-d… (verified 2026-08-21)
    • Perplexity documents the same distinction: PerplexityBot is “designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models” and respects robots.txt; Perplexity-User “Generally ignores robots.txt rules” — docs.perplexity.ai/…/bots (verified 2026-08-21)
    • Google’s robots meta documentation shows the same category separation on the output side — nosnippet/max-snippet gate use “as a direct input for AI Overviews and AI Mode” independently of ordinary indexing — developers.google.com/…/robots-meta-tag (verified 2026-08-21)
    • The other vendors whose tokens the audit counts document the same split (all verified 2026-10-19):
    • Perplexity: PerplexityBot “is not used to crawl content for AI foundation models”. Perplexity-User “generally ignores robots.txt rules” — docs.perplexity.ai/…/perplexity-crawlers
    • Meta: Meta-ExternalAgent crawls “for use cases such as training foundation AI models”. Meta-WebIndexer “navigates the web to improve Meta AI search result quality”. Meta-ExternalFetcher “fetches individual links at a user’s request” — developers.facebook.com/…/web-crawlers
    • Amazon: Amazonbot “may be used to train Amazon AI models”. Amzn-SearchBot and Amzn-User each “does not crawl content for generative AI model training” — developer.amazon.com/amazonbot
    • Mistral: MistralAI-Training builds training datasets. MistralAI-Index content “is not used for generative AI training of any kind”. MistralAI-User serves user actions in Vibe — docs.mistral.ai/robots
    • DuckDuckGo: DuckAssistBot “crawls pages in real-time for our AI-assisted answers” and “This data is not used in any way to train AI models” — duckduckgo.com/…/duckassistbot
    • You.com: “YouBot is the web crawler that powers the You.com search engine” — you.com/…/youbot
    • Allen AI: AI2Bot content “is used to train open language models” — allenai.org/crawler
    • Diffbot: “Diffbot does not crawl pages for training generative AI foundation models” — diffbot.com/…/robots-txt
    • Brave: its crawler “does not advertise a differentiated user agent” — search.brave.com/…/brave-search-crawler
    • Infrastructure has converged on the same taxonomy: Cloudflare’s AI bot controls are grouped as Search, Agent and Training categories, confirming the training/realtime split is an operational reality and not a taxonomy this project invented — developers.cloudflare.com/…/block-ai-bots (verified 2026-08-21)

    Limits

    The A grade covers the capability: separate tokens genuinely receive separate policies. It does not reward the mere presence of granular groups, and no vendor documentation does. A bare User-agent: * with Allow: / grants every named agent the same full access under the RFC 9309 fallback rule (rfc-editor.org/…/rfc9309.html, verified 2026-10-17). The audit therefore returns not applicable on an open catch-all, or on a robots.txt that restricts nothing. Only a catch-all blanket block with no named agents fails, because the fallback carries that block onto the live retrieval agents too.

    Two further limits apply. OpenAI states that for ChatGPT-User, “Because these actions are initiated by a user, robots.txt rules may not apply” (developers.openai.com/…/bots, verified 2026-10-17). Perplexity states that Perplexity-User “generally ignores robots.txt rules” (docs.perplexity.ai/…/bots, verified 2026-10-17). An Allow: / group for user-initiated agents is therefore largely a no-op, and the realtime half of the recommended fix carries less weight than the description implies. Vendor propagation delay is also documented (OpenAI: “it can take ~24 hours from a site’s robots.txt update for our systems to adjust”), so robots.txt state and observed agent behaviour can legitimately diverge at scan time.

    How it scores

    This is a ratified standard (RFC 9309) whose group-matching rule is documented as honored by the named consumers, and OpenAI, Anthropic and Perplexity each publish the training-vs-retrieval token split the audit is built around.

    Sources