Access & Crawl Control
Check whether crawler rules and page responses allow access to your content. Each check includes its proof, limits and effect on the score.
access-crawl-control/agent-governanceGrade AScoredAI crawler vs conversational agent separation
Not all AI bots are the same. Training crawlers like GPTBot, CCBot, and Google-Extended scrape your content to build datasets, while conversational and retrieval agents like ChatGPT-User, Claude-User, and OAI-SearchBot fetch pages live to answer real user questions and can send referral traffic back to you. Many sites want to block the former while welcoming the latter — but a single catch-all User-agent: * cannot express that distinction. Granular robots.txt governance names both categories explicitly so each gets the access policy you actually intend.
access-crawl-control/ai-bot-directivesGrade BScoredAI bot directives are explicit
Reports your robots.txt stance on five long-tail AI bot tokens in one place. Only the bots whose operator publishes crawler documentation — YouBot (You.com) and AI2Bot (Allen Institute) — affect the score, because only those directives have a documented reader. Bytespider, cohere-ai and Diffbot are listed for information: blocking them is a legitimate operational choice that costs no AI-answer visibility.
access-crawl-control/ai-content-declarationGrade DExperimentalAI usage-preference declaration
Where a site declares how AI systems may use its content. The IETF AIPREF work attaches that preference to a Content-Usage response header or a robots.txt rule, and explicitly leaves the HTML head out of scope; the head-level noai/noimageai convention has real adoption but no AI vendor documents honoring it. This audit reports what a site declares and where, and never treats declaring nothing as a defect.
access-crawl-control/ai-crawler-edge-parityGrade AScoredAI crawlers get the same response from the edge that browsers get
Fetches the homepage, sampled content URLs and /llms.txt once as a browser and once per published AI crawler User-Agent, then classifies every difference: Cloudflare challenge, pay-per-crawl 402, proof-of-work wall, rate limit, opaque 403, or a 200 carrying a fraction of the text. Reports per crawler and per URL, and scores a block only where robots.txt said the crawler was welcome.
access-crawl-control/ai-usage-signal-coherence-across-channelsGrade BScoredAI usage signals agree across every channel that carries them
Normalizes every AI-usage signal the site emits — robots.txt Allow/Disallow for training crawlers, AIPREF Content-Usage, Cloudflare Content-Signal, TDM-Rep in its three transports, and inline RSL permits/prohibits — into one comparable model, and reports where two channels contradict each other for the same category over overlapping paths.
access-crawl-control/aipref-content-usage-declaration-validityGrade BScoredAIPREF Content-Usage declarations are valid and can be read
Parses every `Content-Usage` declaration — in robots.txt at file scope and inside each group, and in the response header — as the RFC 8941 dictionary AIPREF defines, validates its categories and values, and checks that the paths it attaches to are actually crawlable. A preference attached to a disallowed path has no effect: attach-05 gives disallowed paths no usage preferences at all.
access-crawl-control/amazonbotGrade AScoredAmazonbot allowed
Without an explicit robots.txt rule, Amazonbot may still crawl your site but has no signal that it is welcome. Adding an explicit allow rule improves your visibility in AI-powered search and ensures consistent crawler behavior.
access-crawl-control/anthropic-aiGrade AScoredClaudeBot crawl access
ClaudeBot collects web content that may contribute to Anthropic's model training, and Anthropic states its bots honour robots.txt. This check reads the robots.txt rules that actually apply to ClaudeBot — its own group if it has one, otherwise the catch-all — and reports whether they let it fetch the site root. A named group is not required: under RFC 9309 §2.2.1 an open catch-all grants the same access. The legacy `anthropic-ai` and `Claude-Web` tokens are reported when present but never scored, because Anthropic's current crawler documentation names neither.
access-crawl-control/applebot-extendedGrade AScoredApplebot-Extended allowed
Without an explicit robots.txt rule, Applebot-Extended may still crawl your site but has no signal that it is welcome. Adding an explicit allow rule improves your visibility in AI-powered search and ensures consistent crawler behavior.
access-crawl-control/bot-content-delta-declaredGrade AScoredContent served to AI crawlers matches the browser, or is declared
Fetches sampled content URLs as a browser and as GPTBot, ClaudeBot and PerplexityBot, then measures the difference two ways — main-text length ratio and 5-gram shingle similarity — because a stub and a rewritten page look identical on length alone. Where a difference exists, requires Google's restricted-content markup and checks that the declared cssSelector resolves against the served DOM.
access-crawl-control/bravebotGrade CAdvisoryBravebot allowed
Without an explicit robots.txt rule, Bravebot may still crawl your site but has no signal that it is welcome. Adding an explicit allow rule improves your visibility in AI-powered search and ensures consistent crawler behavior.
access-crawl-control/canonicalGrade AScoredCanonical URLs point at the right page
A `<link rel="canonical">` tells crawlers which URL is the authoritative version of a page, and the URL they pick is the one eligible to be shown — and cited — in AI answers. The value matters more than the presence: pages that all canonicalize onto the homepage remove themselves from the index.
access-crawl-control/ccbotGrade AScoredCCBot allowed
Without an explicit robots.txt rule, CCBot may still crawl your site but has no signal that it is welcome. Adding an explicit allow rule improves your visibility in AI-powered search and ensures consistent crawler behavior.
access-crawl-control/chatgpt-userGrade CAdvisoryChatGPT-User allowed
Without an explicit robots.txt rule, ChatGPT-User may still crawl your site but has no signal that it is welcome. Adding an explicit allow rule improves your visibility in AI-powered search and ensures consistent crawler behavior.
access-crawl-control/claude-searchbotGrade AScoredClaude-SearchBot allowed
Without an explicit robots.txt rule, Claude-SearchBot may still crawl your site but has no signal that it is welcome. Adding an explicit allow rule improves your visibility in AI-powered search and ensures consistent crawler behavior.
access-crawl-control/claude-userGrade AScoredClaude-User allowed
Without an explicit robots.txt rule, Claude-User may still crawl your site but has no signal that it is welcome. Adding an explicit allow rule improves your visibility in AI-powered search and ensures consistent crawler behavior.
access-crawl-control/crawl-delayGrade CAdvisoryCrawl-delay is reasonable
Excessive Crawl-delay values (over 10 seconds) dramatically slow AI indexing, meaning your latest content may take days or weeks to appear in AI search results.
access-crawl-control/duckassistbotGrade AScoredDuckAssistBot allowed
Without an explicit robots.txt rule, DuckAssistBot may still crawl your site but has no signal that it is welcome. Adding an explicit allow rule improves your visibility in AI-powered search and ensures consistent crawler behavior.
access-crawl-control/google-extendedGrade AScoredGoogle-Extended allowed
Without an explicit robots.txt rule, Google-Extended may still crawl your site but has no signal that it is welcome. Adding an explicit allow rule improves your visibility in AI-powered search and ensures consistent crawler behavior.
access-crawl-control/gptbotGrade AScoredGPTBot allowed
Without an explicit robots.txt rule, GPTBot may still crawl your site but has no signal that it is welcome. Adding an explicit allow rule improves your visibility in AI-powered search and ensures consistent crawler behavior.
access-crawl-control/https-enabledGrade AScoredHTTPS enabled
Enterprise AI frameworks refuse to interact with non-HTTPS sites due to security policies. GPTBot, ClaudeBot, and enterprise RAG systems all skip HTTP-only sites entirely, making your content invisible to AI-generated answers. Enable HTTPS with a valid TLS certificate.
access-crawl-control/machine-actionable-402-paid-accessGrade BScoredA 402 tells a crawler how to pay
Looks at every 402 the crawler-UA probes and the browser baseline received, and asks whether any machine-readable payment mechanism came with it: a `crawler-price` header, an x402 `PAYMENT-REQUIRED` challenge, or an RSL licence with a crawl payment covering the path. A 402 carrying only an HTML page is a price tag no crawler can read.
access-crawl-control/meta-external-agentGrade AScoredMeta-ExternalAgent allowed
Meta-ExternalAgent collects pages for Meta's foundation-model training and for indexing content directly into Meta products, and Meta documents it as respecting robots.txt. This check reads the robots.txt rules that actually apply to it — its own group if it has one, otherwise the catch-all — and reports whether they let it fetch the site root. A named group is not required: under RFC 9309 an open catch-all grants the same access.
access-crawl-control/meta-external-fetcherGrade AScoredMeta-ExternalFetcher allowed
Without an explicit robots.txt rule, Meta-ExternalFetcher may still crawl your site but has no signal that it is welcome. Adding an explicit allow rule improves your visibility in AI-powered search and ensures consistent crawler behavior.
access-crawl-control/mistralai-userGrade AScoredMistralAI-User allowed
Without an explicit robots.txt rule, MistralAI-User may still crawl your site but has no signal that it is welcome. Adding an explicit allow rule improves your visibility in AI-powered search and ensures consistent crawler behavior.
access-crawl-control/no-blanket-blockGrade BScoredNo blanket AI block
A blanket Disallow: / under User-agent: * blocks every crawler, including all AI agents. Your site becomes invisible to AI search engines, ChatGPT Browse, Perplexity, and others.
access-crawl-control/no-bot-detectionGrade AScoredNo aggressive bot-detection blocking agents
Bot-detection services like Cloudflare Turnstile, DataDome, and reCAPTCHA can block legitimate AI agents from accessing your content. Configure your service to allowlist known AI user-agents.
access-crawl-control/no-nofollowGrade AScoredNo nofollow on important links
A site-wide nofollow directive prevents AI crawlers from following links to discover your content. Important internal links should be followable.
access-crawl-control/no-redirect-chainsGrade AScoredNo redirect chains
Redirect chains waste AI crawler budget and slow down content discovery. Each page should resolve in a single redirect at most.
access-crawl-control/oai-searchbotGrade AScoredOAI-SearchBot allowed
Without an explicit robots.txt rule, OAI-SearchBot may still crawl your site but has no signal that it is welcome. Adding an explicit allow rule improves your visibility in AI-powered search and ensures consistent crawler behavior.
access-crawl-control/perplexitybotGrade AScoredPerplexityBot allowed
Without an explicit robots.txt rule, PerplexityBot may still crawl your site but has no signal that it is welcome. Adding an explicit allow rule improves your visibility in AI-powered search and ensures consistent crawler behavior.
access-crawl-control/robots-ai-group-shadowingGrade AScoredrobots.txt AI group shadowing
Detects the RFC 9309 group-precedence trap: adding ANY named group for an AI product token silently voids every rule in the `User-agent: *` group for that bot. Evaluates each AI token twice — under its own merged group and under the wildcard group — with longest-match-wins and Allow-wins-on-tie, and reports three failure classes: a wildcard-protected path reopened for the bot, a named group with no rules at all, and a named group that blocks a bot the wildcard allowed.
access-crawl-control/robots-directivesGrade AScoredRobots directives do not block AI indexing
Robots directives — `<meta name="robots">`, per-bot meta tags and the `X-Robots-Tag` header — decide whether a page can be indexed at all and whether it can be quoted in an AI answer. A `noindex` on a content page removes it from the index AI Overviews and AI Mode draw on; `nosnippet` and `max-snippet:0` leave it indexed but unquotable.
access-crawl-control/rsl-licensing-terms-conformanceGrade BScoredRSL licensing terms are discoverable and conformant
Looks for an RSL licence in all four channels the specification defines — the robots.txt `License:` directive, a `Link: rel=license` response header, an HTML `<link>`, and an inline `<script type="application/rsl+xml">` — then validates the document: its namespace, its media type, the paths its `<content url>` covers, and the enumerated attributes on every permits, prohibits, payment and amount element.
access-crawl-control/sensitive-pathsGrade AScoredLow-value URLs excluded from AI crawls
AI crawlers honour path-level Disallow rules (RFC 9309), so robots.txt is the lever for keeping low-value URL spaces — carts, checkouts, site-search results, login and account pages — out of AI crawls and out of the answers built from them. This is crawl hygiene, not access control.
access-crawl-control/tdm-repGrade CExperimentalTDM-Rep declaration
TDM-Rep is a W3C Community Group Final Report — explicitly not a W3C Standard — that defines a machine-readable text-and-data-mining reservation, anchored in EU DSM Directive Article 4. Its value is legal evidence of an opt-out, not agent behaviour: no major AI crawler operator documents honouring it. This audit reports what a site declares and in which direction, and never scores it either way.
access-crawl-control/web-bot-auth-request-toleranceGrade BScoredA signed agent request is not rejected for being signed
Fetches the site root twice: once plainly, and once with the RFC 9421 `Signature`, `Signature-Input` and `Signature-Agent` headers a Web Bot Auth agent sends, signed with a per-scan ephemeral Ed25519 key. Reports when the signed request is refused, truncated, or answered differently without `Vary` naming the headers that changed the answer.