Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/ccbot

    CCBot allowed

    What it checks

    Reads the robots.txt rules that apply to CCBot — its own group if it has one, otherwise the catch-all — and reports whether they let it fetch the site root. A named group is not required: under RFC 9309 §2.2.1 an open catch-all grants the same access.

    Why it matters

    Disallowing CCBot keeps the site out of the Common Crawl corpus. That corpus is the upstream source for C4, RefinedWeb and Dolma, and therefore for many LLM training sets. A block here has downstream training effects far beyond one operator.

    Evidence

    CCBot allow/block state in robots.txt

    Common Crawl publishes UA ‘CCBot/2.0 (commoncrawl.org/faq)’ and the canonical opt-out snippet ‘User-agent: CCBot / Disallow: /’. The training-corpus leverage is quantified by the Data Provenance Initiative’s ‘Consent in Crisis’, which audited 14,000 domains. Within a single year (2023-2024), robots.txt restrictions rendered ‘~5%+ of all tokens in C4, or 28%+ of the most actively maintained, critical sources in C4, fully restricted from use’. CCBot is therefore the highest-leverage single token for training-data opt-out.

    Limits

    Blocking CCBot is retroactively useless — historical Common Crawl snapshots are already published and permanently redistributable, so a block only affects future crawls. Common Crawl warns that ‘crawlers falsely identifying themselves as CCBot’ exist, so a disallow does not stop spoofers; operators should verify by reverse DNS against published IP ranges. Common Crawl’s own page states no crawl-delay position and does not frame itself as AI training infrastructure.

    How it scores

    Common Crawl publishes the exact user agent and the canonical opt-out snippet. The downstream leverage is measured rather than asserted. The Data Provenance Initiative’s “Consent in Crisis” audited 14,000 domains. Within a single year, robots.txt restrictions had rendered “~5%+ of all tokens in C4, or 28%+ of the most actively maintained, critical sources in C4, fully restricted from use”. A documented token with a measured corpus effect is grade A.

    Sources