Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/bravebot

    Bravebot allowed

    What it checks

    Without an explicit robots.txt rule, Bravebot may still crawl your site but has no signal that it is welcome. Adding an explicit allow rule improves your visibility in AI-powered search and ensures consistent crawler behavior.

    Why it matters

    A ‘Bravebot’ disallow is intended to block Brave Search’s crawler. But Brave’s own documentation states its crawler deliberately does not advertise a differentiated user agent. The token therefore has no vendor-confirmed consumer and the rule is likely a no-op.

    Evidence

    BraveBot

    Known Agents lists a Bravebot entry with UA ‘Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Bravebot/1.0; +https://search.brave.com/help/brave-search-crawler) Chrome/W.X.Y.Z Safari/537.36’ and only 2% top-website blocking as of 2026-08-19 — the lowest of any token in this set, indicating near-zero operator recognition.

    Limits

    The vendor refutes this directly: Brave’s own crawler help page states ‘The Brave Search crawler does not advertise a differentiated user agent because we must avoid discrimination from websites that allow only Google to crawl them.’ Brave further states ‘robots.txt is not used to prevent a page from being indexed. A site owner can delist a page by using the robots noindex directive’ — i.e. Brave directs publishers to noindex, not to a robots.txt token.

    Brave’s page makes no mention of AI training, data licensing, or Brave Leo in connection with the crawler. Given the vendor contradicts the token’s existence and blocking adoption is 2%, this should never be scored; consider demoting the audit toward deletion unless a Brave-published token is confirmed.

    Sources