Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/oai-searchbot

    OAI-SearchBot allowed

    What it checks

    Reads the robots.txt rules that apply to OAI-SearchBot — its own group if it has one, otherwise the catch-all — and reports whether they let it fetch the site root. A named group is not required: under RFC 9309 §2.2.1 an open catch-all grants the same access.

    Why it matters

    Disallowing OAI-SearchBot causes the site to be excluded from ChatGPT search answers. This is the single strongest documented allow-side signal in the whole domain: blocking it directly destroys AI answer visibility.

    Evidence

    OAI-SearchBot allow/block state in robots.txt

    OpenAI documents this as a direct causal consequence: ‘Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers.’ UA ‘OAI-SearchBot/1.4; +https://openai.com/searchbot’, IPs at openai.com/searchbot.json. This is a vendor-stated, falsifiable behavioral claim, not an inference — grade A, and it justifies scoring an OAI-SearchBot disallow as a negative for any site that wants AI answer visibility.

    Limits

    BuzzStream/XOFU measured 82.4% citation retention even among sites blocking OAI-SearchBot, suggesting either lagged enforcement, citation via other surfaces, or content reached through ChatGPT-User. The vendor claim and the field measurement are in tension; the vendor claim is the stated policy and should be scored, but the audit copy should not promise total disappearance.

    How it scores

    This is the strongest allow-side statement in the category, and it is causal rather than descriptive: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers.” OpenAI names the token, publishes the user agent and an IP list, and states the consequence itself. That is well past the grade-A bar. One tension is on the record: an independent measurement found 82.4% citation retention among sites blocking the token, which suggests lagged enforcement or citation through other surfaces. The vendor’s stated policy is what the audit scores, and the copy states the measurement rather than hiding it.

    Sources