Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/gptbot

    GPTBot allowed

    What it checks

    Reads the robots.txt rules that apply to GPTBot — its own group if it has one, otherwise the catch-all — and reports whether they let it fetch the site root. A named group is not required: under RFC 9309 §2.2.1 an open catch-all grants the same access.

    Why it matters

    Disallowing GPTBot removes the site’s content from OpenAI foundation-model training corpora; allowing it permits training use. It does not affect ChatGPT search visibility (that is OAI-SearchBot).

    Evidence

    GPTBot allow/block state in robots.txt

    OpenAI’s bot documentation states verbatim: ‘Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.’ UA ‘GPTBot/1.4; +https://openai.com/gptbot’, IP ranges published at openai.com/gptbot.json for verification. Activity confirmed at network scale: Cloudflare Radar found ClaudeBot and GPTBot together ‘account for nearly half of the observed crawling activity’ (Aug 2025), and GPTBot was 17.4% of AI crawl traffic in News & Publications. Documented ACTIVE in 2026.

    Limits

    Blocking GPTBot has a measurably small effect on downstream visibility: BuzzStream/XOFU found sites blocking GPTBot still retained 88.2% citation presence in AI answers, because already-ingested and third-party-mirrored content persists. Consent in Crisis found OpenAI is the single most-blocked developer, so a GPTBot block is not differentiating. Auditors must not conflate a GPTBot block with an OAI-SearchBot block — they have opposite visibility consequences.

    How it scores

    OpenAI states the effect of the directive verbatim: “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.” The token, the user agent and a published IP list for verification all come from the vendor, which is the grade-A bar. Two things the grade does not cover, and the audit’s copy says so. Blocking GPTBot has only a small measured effect on downstream visibility: sites blocking it retained 88.2% citation presence, because ingested and mirrored content persists. And GPTBot is not a ChatGPT search control — that is OAI-SearchBot.

    Sources