Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/chatgpt-user

    ChatGPT-User robots.txt group (may not apply)

    What it checks

    Whether robots.txt has a group that names ChatGPT-User. A catch-all group that happens to cover the token is not reported: this audit speaks only about a group the owner wrote for ChatGPT-User.

    State Result
    a User-agent: ChatGPT-User group, allowing or blocking pass
    no robots.txt, a soft 404, or no group naming ChatGPT-User notApplicable

    The pass message quotes OpenAI: the group may not be honoured, and OAI-SearchBot is the token for search opt-outs. Matching is case-insensitive and ignores a version suffix such as ChatGPT-User/1.0.

    Why it matters

    Mechanism: a named robots.txt group controls the crawler that sends that product token. The claim this audit tests is that OpenAI does not commit to obeying the group for ChatGPT-User, so it is not a reliable control.

    Evidence

    • OpenAI: “ChatGPT-User is not used for crawling the web in an automatic fashion. Because these actions are initiated by a user, robots.txt rules may not apply. … Please use OAI-SearchBot in robots.txt for managing Search opt outs and automatic crawl.” — Overview of OpenAI Crawlers (verified 2026-10-18)

    Limits

    “May not apply” is not “never applies”. The audit says the group may not be honoured, not that it is ignored.

    How it scores

    Grade: C — OpenAI says robots.txt rules “may not apply” to ChatGPT-User. That is a hedge, not a documented refusal, so the group is informative with an unconsumed notice rather than obsolete (docs/evidence/policy.md, “Informative or obsolete”). The audit carries weight 0, never warns or fails, and does not move any score.

    Example finding

    robots.txt has User-agent: ChatGPT-User / Disallow: /. The report says the rule may not be honoured and names OAI-SearchBot for search opt-outs.

    Sources