Access & Crawl Control
Check whether crawler rules and page responses allow access to your content. Each check includes its proof, limits and effect on the score.
access-crawl-control/agent-governanceGrade AScoredAI crawler vs conversational agent separation
Not all AI bots are the same. Training crawlers like GPTBot, CCBot, and Google-Extended scrape your content to build datasets, while conversational and retrieval agents like ChatGPT-User, Claude-User, and OAI-SearchBot fetch pages live to answer real user questions and can send referral traffic back to you. Many sites want to block the former while welcoming the latter — but a single catch-all User-agent: * cannot express that distinction. Granular robots.txt governance names both categories explicitly so each gets the access policy you actually intend.
access-crawl-control/ai-bot-directivesGrade BScoredDocumented AI bots allowed by robots.txt
Reports your robots.txt stance on five long-tail AI bot tokens in one place. Only the bots whose operator publishes crawler documentation — YouBot (You.com) and AI2Bot (Allen Institute) — affect the score, because only those directives have a documented reader. Bytespider, cohere-ai and Diffbot are listed for information: blocking them is a legitimate operational choice that costs no AI-answer visibility.
access-crawl-control/ai-content-declarationGrade DExperimentalAI usage-preference declaration
Where a site declares how AI systems may use its content. The IETF AIPREF work attaches that preference to a Content-Usage response header or a robots.txt rule, and explicitly leaves the HTML head out of scope; the head-level noai/noimageai convention has real adoption but no AI vendor documents honoring it. This audit reports what a site declares and where, and never treats declaring nothing as a defect.
access-crawl-control/ai-crawler-edge-parityGrade AScoredRequests naming an AI crawler get the same edge response as a browser
Fetches the homepage, sampled content URLs and /llms.txt once as a browser and once per published AI crawler User-Agent, then classifies every difference: Cloudflare challenge, pay-per-crawl 402, proof-of-work wall, rate limit, opaque 403, or a 200 carrying a fraction of the text. Reports per crawler and per URL, and scores a difference only where robots.txt admits that crawler: a pay-per-crawl price or a reduced page fails, while a challenge or another refusal warns, because an edge can refuse an unverified request naming the crawler and still admit the real one. The probes come from the scanner's address, so they show how the site answers a request that names the crawler, not what the real crawler, verified by IP range or signature, receives.
access-crawl-control/ai-usage-signal-coherence-across-channelsGrade BScoredAI usage signals agree across every channel that carries them
Normalizes every AI-usage signal the site emits — robots.txt Allow/Disallow for training crawlers, AIPREF Content-Usage, Cloudflare Content-Signal, TDM-Rep in its three transports, and inline RSL permits/prohibits — into one comparable model, and reports where two channels contradict each other for the same category over overlapping paths.
access-crawl-control/aipref-content-usage-declaration-validityGrade BScoredAIPREF Content-Usage declarations are valid and can be read
Parses every `Content-Usage` declaration — in robots.txt at file scope and inside each group, and in the response header — as the RFC 8941 dictionary AIPREF defines, validates its categories and values, and checks that the paths it attaches to are actually crawlable. A preference attached to a disallowed path has no effect: attach-05 gives disallowed paths no usage preferences at all.
access-crawl-control/amazonbotGrade AScoredAmazonbot allowed
Reads the robots.txt rules that apply to Amazonbot — its own group if it has one, otherwise the catch-all — and reports whether they let it fetch the site root. A named group is not required: under RFC 9309 §2.2.1 an open catch-all grants the same access.
access-crawl-control/anthropic-aiGrade AScoredClaudeBot crawl access
ClaudeBot collects web content that may contribute to Anthropic's model training, and Anthropic states its bots honour robots.txt. This check reads the robots.txt rules that actually apply to ClaudeBot — its own group if it has one, otherwise the catch-all — and reports whether they let it fetch the site root. A named group is not required: under RFC 9309 §2.2.1 an open catch-all grants the same access. The legacy `anthropic-ai` and `Claude-Web` tokens are reported when present but never scored, because Anthropic's current crawler documentation names neither.
access-crawl-control/applebot-extendedGrade AScoredApplebot-Extended allowed
Applebot-Extended is a robots.txt usage-control token, not a crawler: no request carries it. This check reads the rules that apply to it — its own group if it has one, otherwise the catch-all — and reports whether they disallow the site root. A named group is not required: under RFC 9309 §2.2.1 an open catch-all grants the same permission.
access-crawl-control/bot-content-delta-declaredGrade AScoredContent served to AI crawler User-Agents matches the browser, or is declared
Fetches sampled content URLs as a browser and with the GPTBot, ClaudeBot and PerplexityBot User-Agents, then measures the difference two ways — main-text length ratio and 5-gram shingle similarity — because a stub and a rewritten page look identical on length alone. Where a difference exists, requires Google's restricted-content markup and checks that the declared cssSelector resolves against the served DOM. A delta an unrecognised control bot gets as well is bot management aimed at unidentified clients, not a page keyed on the crawler's name, and warns.
access-crawl-control/bravebotGrade DObsoleteBravebot robots.txt group (unread)
Reports a `User-agent: Bravebot` group in robots.txt. Brave documents that its search crawler does not advertise a distinct user agent and follows the rules that apply to Googlebot. It does not document reading a Bravebot group.
access-crawl-control/canonicalGrade AScoredCanonical URLs point at the right page
A `<link rel="canonical">` tells crawlers which URL is the authoritative version of a page, and the URL they pick is the one eligible to be shown — and cited — in AI answers. The value matters more than the presence: pages that all canonicalize onto the homepage remove themselves from the index.
access-crawl-control/ccbotGrade AScoredCCBot allowed
Reads the robots.txt rules that apply to CCBot — its own group if it has one, otherwise the catch-all — and reports whether they let it fetch the site root. A named group is not required: under RFC 9309 §2.2.1 an open catch-all grants the same access.
access-crawl-control/chatgpt-userGrade CAdvisoryChatGPT-User robots.txt group (may not apply)
Reports a `User-agent: ChatGPT-User` group in robots.txt. OpenAI documents that ChatGPT-User fetches pages on a user's request, that robots.txt rules may not apply to those fetches, and that OAI-SearchBot is the token for managing search opt-outs.
access-crawl-control/claude-searchbotGrade AScoredClaude-SearchBot allowed
Reads the robots.txt rules that apply to Claude-SearchBot — its own group if it has one, otherwise the catch-all — and reports whether they let it fetch the site root. A named group is not required: under RFC 9309 §2.2.1 an open catch-all grants the same access.
access-crawl-control/claude-userGrade AScoredClaude-User allowed
Reads the robots.txt rules that apply to Claude-User — its own group if it has one, otherwise the catch-all — and reports whether they let it fetch the site root. A named group is not required: under RFC 9309 §2.2.1 an open catch-all grants the same access.
access-crawl-control/content-signalsGrade CAdvisoryContent Signals choices and optional setup
Explains declared search, AI-input and AI-training preferences by bot and path, with syntax findings and optional client choices.
access-crawl-control/crawl-delayGrade CAdvisoryCrawl-delay is reasonable
Checks the Crawl-delay values robots.txt sets for crawlers that honour the directive (Anthropic's crawlers, Bingbot, CCBot, YouBot). Bing recommends no value above 10 seconds. Groups addressed only to Google, Amazonbot or Applebot are not judged: those crawlers document that they ignore Crawl-delay.
access-crawl-control/duckassistbotGrade AScoredDuckAssistBot allowed
Reads the robots.txt rules that apply to DuckAssistBot — its own group if it has one, otherwise the catch-all — and reports whether they let it fetch the site root. A named group is not required: under RFC 9309 §2.2.1 an open catch-all grants the same access.
access-crawl-control/google-extendedGrade AScoredGoogle-Extended allowed
Google-Extended is a robots.txt usage-control token, not a crawler: no request carries it. This check reads the rules that apply to it — its own group if it has one, otherwise the catch-all — and reports whether they disallow the site root. A named group is not required: under RFC 9309 §2.2.1 an open catch-all grants the same permission.
access-crawl-control/gptbotGrade AScoredGPTBot allowed
Reads the robots.txt rules that apply to GPTBot — its own group if it has one, otherwise the catch-all — and reports whether they let it fetch the site root. A named group is not required: under RFC 9309 §2.2.1 an open catch-all grants the same access.
access-crawl-control/https-enabledGrade BScoredHTTPS enabled
Since Chrome 154 (stable 2026-09-22), Chrome prompts users by default before it opens a site over plain http, so an agent working inside Chrome meets that prompt before any content. No agent vendor documents how its agent handles the prompt, and no AI crawler vendor documents refusing plain HTTP, so the risk is to agents that act in a browser, not to crawling. Enable HTTPS with a valid TLS certificate.
access-crawl-control/machine-actionable-402-paid-accessGrade BScoredA 402 tells a crawler how to pay
Looks at every 402 the crawler-UA probes and the browser baseline received, and asks whether any machine-readable payment mechanism came with it: a `crawler-price` header, an x402 `PAYMENT-REQUIRED` challenge, or an RSL licence with a crawl payment covering the path. A 402 carrying only an HTML page is a price tag no crawler can read. The probes come from the scanner's address and carry no Web Bot Auth signature, so they read the 402 a request naming the crawler gets; they never exercise the paid path.
access-crawl-control/meta-external-agentGrade AScoredMeta-ExternalAgent allowed
Meta-ExternalAgent collects pages for Meta's foundation-model training and for indexing content directly into Meta products, and Meta documents it as respecting robots.txt. This check reads the robots.txt rules that actually apply to it — its own group if it has one, otherwise the catch-all — and reports whether they let it fetch the site root. A named group is not required: under RFC 9309 an open catch-all grants the same access.
access-crawl-control/meta-external-fetcherGrade AScoredMeta-ExternalFetcher allowed
Reads the robots.txt rules that apply to Meta-ExternalFetcher — its own group if it has one, otherwise the catch-all — and reports whether they let it fetch the site root. A named group is not required: under RFC 9309 §2.2.1 an open catch-all grants the same access.
access-crawl-control/mistralai-userGrade AScoredMistralAI-User allowed
Reads the robots.txt rules that apply to MistralAI-User — its own group if it has one, otherwise the catch-all — and reports whether they let it fetch the site root. A named group is not required: under RFC 9309 §2.2.1 an open catch-all grants the same access.
access-crawl-control/no-blanket-blockGrade BScoredNo blanket AI block
A blanket Disallow: / under User-agent: * blocks every crawler, including all AI agents. Your site becomes invisible to AI search engines, ChatGPT Browse, Perplexity, and others.
access-crawl-control/no-bot-detectionGrade AScoredNo aggressive bot-detection blocking agents
Bot-detection services like Cloudflare Turnstile, DataDome, and reCAPTCHA can block legitimate AI agents from accessing your content. Configure your service to allowlist known AI user-agents.
access-crawl-control/no-nofollowGrade AScoredPage-level nofollow directives
Checks for page-level nofollow in robots metadata and X-Robots-Tag headers across the scanned pages. This check does not inspect individual links.
access-crawl-control/no-redirect-chainsGrade AScoredNo redirect chains
Redirect chains waste AI crawler budget and slow down content discovery. Each page should resolve in a single same-site redirect at most.
access-crawl-control/oai-searchbotGrade AScoredOAI-SearchBot allowed
Reads the robots.txt rules that apply to OAI-SearchBot — its own group if it has one, otherwise the catch-all — and reports whether they let it fetch the site root. A named group is not required: under RFC 9309 §2.2.1 an open catch-all grants the same access.
access-crawl-control/perplexitybotGrade AScoredPerplexityBot allowed
Reads the robots.txt rules that apply to PerplexityBot — its own group if it has one, otherwise the catch-all — and reports whether they let it fetch the site root. A named group is not required: under RFC 9309 §2.2.1 an open catch-all grants the same access.
access-crawl-control/robots-ai-group-shadowingGrade AScoredrobots.txt AI group shadowing
Detects the RFC 9309 group-precedence trap: adding ANY named group for an AI product token silently voids every rule in the `User-agent: *` group for that bot. Evaluates each AI token twice — under its own merged group and under the wildcard group — with longest-match-wins and Allow-wins-on-tie, and reports three failure classes: a wildcard-protected path reopened for the bot, a named group with no rules at all, and a named group that blocks a bot the wildcard allowed.
access-crawl-control/robots-directivesGrade AScoredRobots directives do not block AI indexing
Robots directives — `<meta name="robots">`, per-bot meta tags and the `X-Robots-Tag` header — decide whether a page can be indexed at all and whether it can be quoted in an AI answer. A `noindex` on a non-utility page removes it from the index AI Overviews and AI Mode draw on; `nosnippet` and `max-snippet:0` leave it indexed but unquotable.
access-crawl-control/rsl-licensing-terms-conformanceGrade BScoredRSL licensing terms are discoverable and conformant
Looks for an RSL licence in all four channels the specification defines — the robots.txt `License:` directive, a `Link: rel=license` response header, an HTML `<link>`, and an inline `<script type="application/rsl+xml">` — then validates the document: its namespace, its media type, the paths its `<content url>` covers, and the enumerated attributes on every permits, prohibits, payment and amount element.
access-crawl-control/sensitive-pathsGrade AScoredLow-value URLs excluded from AI crawls
AI crawlers honour path-level Disallow rules (RFC 9309), so robots.txt is the lever for keeping low-value URL spaces — carts, checkouts, site-search results, login and account pages — out of AI crawls and out of the answers built from them. This is crawl hygiene, not access control.
access-crawl-control/tdm-repGrade CAdvisoryTDM-Rep declaration
TDM-Rep is a W3C Community Group Final Report — explicitly not a W3C Standard — that defines a machine-readable text-and-data-mining reservation, anchored in EU DSM Directive Article 4. Its value is legal evidence of an opt-out, not agent behaviour: no major AI crawler operator documents honouring it. This audit reports what a site declares and in which direction, and never scores it either way.
access-crawl-control/web-bot-auth-request-toleranceGrade BScoredA signed agent request is not rejected for being signed
Fetches the site root twice: once plainly, and once with the RFC 9421 `Signature`, `Signature-Input` and `Signature-Agent` headers a Web Bot Auth agent sends, signed with a per-scan ephemeral Ed25519 key. Reports when the signed request is refused, truncated, or answered differently without `Vary` naming the headers that changed the answer.