Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/agent-governance

    AI crawler vs conversational agent separation

    What it checks

    Not all AI bots are the same. Training crawlers like GPTBot, CCBot, and Google-Extended scrape your content to build datasets, while conversational and retrieval agents like ChatGPT-User, Claude-User, and OAI-SearchBot fetch pages live to answer real user questions and can send referral traffic back to you. Many sites want to block the former while welcoming the latter — but a single catch-all User-agent: * cannot express that distinction. Granular robots.txt governance names both categories explicitly so each gets the access policy you actually intend.

    Why it matters

    Each major AI vendor operates separate robots.txt product tokens for dataset-training crawling and for live retrieval or search grounding. Per RFC 9309, a crawler obeys the group matching its own token, and falls back to * only when no such group exists. A robots.txt that names the two categories separately therefore produces different access outcomes for training and for live retrieval — an outcome a catch-all group alone cannot express.

    Evidence

    • RFC 9309 §2.2.1: “Crawlers MUST use case-insensitive matching to find the group that matches the product token and then obey the rules of the group”. If no specific match exists, “crawlers MUST obey the group with a user-agent line with the ‘*’ value, if present”. A specific group therefore overrides the catch-all for that agent — the exact capability the audit measures — rfc-editor.org/…/rfc9309.html (verified 2026-08-21)
    • OpenAI documents the split directly. GPTBot is “used to make our generative AI foundation models more useful and safe”, and “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models”. OAI-SearchBot is “used to surface websites in search results in ChatGPT’s search features”, and “we recommend allowing OAI-SearchBot in your site’s robots.txt file”. ChatGPT-User is user-initiated — developers.openai.com/…/bots (verified 2026-08-21)
    • Anthropic operates three tokens with distinct purposes — ClaudeBot (training-corpus collection), Claude-User (user-initiated fetches), Claude-SearchBot (search-quality analysis) — and documents per-agent robots.txt groups (User-agent: ClaudeBot / Disallow: /) — support.claude.com/…/8896518-does-anthropic-crawl-d… (verified 2026-08-21)
    • Perplexity documents the same distinction: PerplexityBot is “designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models” and respects robots.txt; Perplexity-User “Generally ignores robots.txt rules” — docs.perplexity.ai/…/bots (verified 2026-08-21)
    • Google’s robots meta documentation shows the same category separation on the output side — nosnippet/max-snippet gate use “as a direct input for AI Overviews and AI Mode” independently of ordinary indexing — developers.google.com/…/robots-meta-tag (verified 2026-08-21)
    • Infrastructure has converged on the same taxonomy: Cloudflare’s AI bot controls are grouped as Search, Agent and Training categories, confirming the training/realtime split is an operational reality and not a taxonomy this project invented — developers.cloudflare.com/…/block-ai-bots (verified 2026-08-21)

    Limits

    The A grade covers the capability — separate tokens genuinely receive separate policies. It does not support the audit’s pass criterion. No vendor documentation rewards the mere presence of granular groups. A bare User-agent: * with Allow: / grants every named agent identical full access under the RFC 9309 fallback rule. The current FAIL on that configuration therefore contradicts the cited standard. Two further limits apply. OpenAI states that for ChatGPT-User, “Because these actions are initiated by a user, robots.txt rules may not apply”.

    Perplexity states that Perplexity-User “Generally ignores robots.txt rules”. An Allow: / group welcoming user-initiated agents is therefore largely a no-op, and the realtime half of the recommended fix carries less weight than the description implies. Vendor propagation delay is also documented (OpenAI: “it can take ~24 hours from a site’s robots.txt update for our systems to adjust”), so robots.txt state and observed agent behavior can legitimately diverge at scan time.

    How it scores

    This is a ratified standard (RFC 9309) whose group-matching rule is documented as honored by the named consumers, and OpenAI, Anthropic and Perplexity each publish the training-vs-retrieval token split the audit is built around.

    Sources