Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/bravebot

    Bravebot robots.txt group (unread)

    What it checks

    Whether robots.txt has a group that names Bravebot. A catch-all group that happens to cover the token is not reported: this audit speaks only about a group the owner wrote for Brave.

    State Result
    a User-agent: Bravebot group, allowing or blocking warn, priority low
    no robots.txt, a soft 404, or no group naming Bravebot notApplicable

    Why it matters

    Mechanism: a named robots.txt group controls the crawler that sends that product token. The claim this audit tests is that Brave documents following the Googlebot rules and does not document reading a Bravebot group, so the group is not the control Brave describes.

    Evidence

    • Brave: “The Brave Search crawler does not advertise a differentiated user agent because we must avoid discrimination from websites that allow only Google to crawl them. However, if a domain or page is not crawlable by Googlebot, then Brave Search’s bot will not crawl it either.” — Brave Search Crawler (verified 2026-10-18)

    Limits

    Third-party bot directories list a Bravebot/1.0 user-agent string. Brave does not document one, and its page does not say a Bravebot group is ignored. The message therefore says only what Brave documents: it follows the Googlebot rules.

    How it scores

    Grade: D — the operator documents following the Googlebot rules and sending no distinct user agent; it documents no reading of a Bravebot group. An obsolete audit carries weight 0 and never returns fail (sunset.test.ts; Audit.toCheckResult lowers a stray fail to warn). It does not move any score.

    Example finding

    robots.txt has User-agent: Bravebot / Disallow: /private. The report says Brave documents following the Googlebot rules and does not document reading a Bravebot group.

    Sources