Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/sensitive-paths

    Low-value URLs excluded from AI crawls

    What it checks

    Whether the low-value URL families a site actually has are kept out of AI crawls by robots.txt. This is crawl hygiene, not security. RFC 9309 states that robots.txt “is not a substitute for valid content security measures”, and anything that needs protecting belongs behind HTTP authentication.

    The audit looks for five families: cart and checkout, site search, authentication, account areas and admin surfaces. It finds them in the crawled page URLs, the same-origin links on those pages and the sitemap. A family the crawl never observed is never asked for. API paths are not a low-value family. Agents need them, and the audit never asks a site to disallow them.

    Each observed path is matched with RFC 9309 rules against six AI crawler tokens: GPTBot, ClaudeBot, Applebot-Extended, meta-externalagent, Google-Extended and PerplexityBot. A path counts as excluded only when all six are disallowed from it.

    State Result
    no low-value family observed na
    robots.txt blocks all six crawlers at the root na — no-blanket-block owns that finding
    every observed family excluded pass
    some families excluded, others still crawlable warn, priority low
    no observed family excluded, or no robots.txt is served fail, priority low

    A warn or fail carries the Disallow lines that would make the audit pass.

    Why it matters

    AI crawlers apply RFC 9309 path matching, not only a site-level allow or block. A Disallow naming a directory therefore keeps the URLs under it out of the crawls those agents perform. That is what makes a per-path rule a usable crawl-hygiene instrument rather than a decorative one.

    Evidence

    path-level robots.txt Disallow honoured by named AI crawlers

    • RFC 9309 is the ratified standard, and its matching rule is normative: “The most specific match found MUST be used” — rfc-editor.org/…/rfc9309.html (verified 2026-08-21)
    • Apple documents the rule for both of its tokens with a literal directory. For the ordinary crawler: “Applebot doesn’t try to crawl documents that are under /private/ or /not-allowed/”. For the generative-AI training token: “You can add a rule in robots.txt to disallow Applebot-Extended, as follows: User-agent: Applebot-Extended / Disallow: /private/” — support.apple.com/…/119829 (verified 2026-08-21)
    • Meta documents the same shape for a crawler that “crawls the web for use cases such as training foundation AI models”: “User-agent: meta-externalagent / Allow: / # Allow everything / Disallow: /private/ # Disallow a specific directory” — developers.facebook.com/…/web-crawlers (verified 2026-08-21)
    • Anthropic states that “Anthropic’s Bots respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt” for ClaudeBot, Claude-User and Claude-SearchBot, though its own examples are root-level only — support.claude.com/…/8896518-does-anthropic-crawl-d… (verified 2026-08-21)
    • OpenAI directs publishers to robots.txt for GPTBot and OAI-SearchBot opt-outs — developers.openai.com/…/bots (verified 2026-08-21)

    Limits

    The mechanism is real; the audit’s original benefit — security and privacy, keeping internal endpoints out of training data — is contradicted by the standard it rests on. RFC 9309 states outright that “The Robots Exclusion Protocol is not a substitute for valid content security measures”, and warns that listing paths makes them publicly discoverable, directing operators to HTTP authentication instead. The old guidance therefore told site owners to publish a map of their admin surface.

    The agent traffic most likely to reach /admin/ or /api/ is also exempt. OpenAI states that for ChatGPT-User, “Because these actions are initiated by a user, robots.txt rules may not apply”. Perplexity states that Perplexity-User “generally ignores robots.txt rules” (docs.perplexity.ai/…/bots, verified 2026-08-21). Finally there is a domain-fit limit that shaped the rewrite: /api/ is precisely the surface an agent wants, so a high-priority failure telling every site to disallow it works against the outcome this project exists to improve.

    All four points are why the audit was rewritten to crawl hygiene over observed URL families, at low priority, rather than kept as a security check.

    How it scores

    Two AI vendors document path-level Disallow for their own AI crawlers with literal directory examples, on top of a ratified standard whose matching rule they implement. That is documented consumer behaviour for exactly the signal this audit inspects.

    Sources