Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/gptbot

    GPTBot allowed

    What it checks

    Without an explicit robots.txt rule, GPTBot may still crawl your site but has no signal that it is welcome. Adding an explicit allow rule improves your visibility in AI-powered search and ensures consistent crawler behavior.

    Why it matters

    Disallowing GPTBot removes the site’s content from OpenAI foundation-model training corpora; allowing it permits training use. It does not affect ChatGPT search visibility (that is OAI-SearchBot).

    Evidence

    GPTBot allow/block state in robots.txt

    OpenAI’s bot documentation states verbatim: ‘Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.’ UA ‘GPTBot/1.4; +https://openai.com/gptbot’, IP ranges published at openai.com/gptbot.json for verification. Activity confirmed at network scale: Cloudflare Radar found ClaudeBot and GPTBot together ‘account for nearly half of the observed crawling activity’ (Aug 2025), and GPTBot was 17.4% of AI crawl traffic in News & Publications. Documented ACTIVE in 2026.

    Limits

    Blocking GPTBot has a measurably small effect on downstream visibility: BuzzStream/XOFU found sites blocking GPTBot still retained 88.2% citation presence in AI answers, because already-ingested and third-party-mirrored content persists. Consent in Crisis found OpenAI is the single most-blocked developer, so a GPTBot block is not differentiating. Auditors must not conflate a GPTBot block with an OAI-SearchBot block — they have opposite visibility consequences.

    How it scores

    OpenAI states the effect of the directive verbatim: “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.” The token, the user agent and a published IP list for verification all come from the vendor, which is the grade-A bar. Two things the grade does not cover, and the audit’s copy says so. Blocking GPTBot has only a small measured effect on downstream visibility: sites blocking it retained 88.2% citation presence, because ingested and mirrored content persists. And GPTBot is not a ChatGPT search control — that is OAI-SearchBot.

    Sources