Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/sensitive-paths

    Low-value URLs excluded from AI crawls

    What it checks

    Without robots.txt, AI crawlers can access sensitive paths like /api/ and /admin/. This may expose internal endpoints, admin panels, or debug information in AI training data and search results.

    Why it matters

    AI crawlers apply RFC 9309 path matching, not only a site-level allow or block. A Disallow naming a directory therefore keeps the URLs under it out of the crawls those agents perform. That is what makes a per-path rule a usable crawl-hygiene instrument rather than a decorative one.

    Evidence

    path-level robots.txt Disallow honoured by named AI crawlers

    • RFC 9309 is the ratified standard, and its matching rule is normative: “The most specific match found MUST be used” — rfc-editor.org/…/rfc9309.html (verified 2026-08-21)
    • Apple documents the rule for both of its tokens with a literal directory. For the ordinary crawler: “Applebot doesn’t try to crawl documents that are under /private/ or /not-allowed/”. For the generative-AI training token: “You can add a rule in robots.txt to disallow Applebot-Extended, as follows: User-agent: Applebot-Extended / Disallow: /private/” — support.apple.com/…/119829 (verified 2026-08-21)
    • Meta documents the same shape for a crawler that “crawls the web for use cases such as training foundation AI models”: “User-agent: meta-externalagent / Allow: / # Allow everything / Disallow: /private/ # Disallow a specific directory” — developers.facebook.com/…/web-crawlers (verified 2026-08-21)
    • Anthropic states that “Anthropic’s Bots respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt” for ClaudeBot, Claude-User and Claude-SearchBot, though its own examples are root-level only — support.claude.com/…/8896518-does-anthropic-crawl-d… (verified 2026-08-21)
    • OpenAI directs publishers to robots.txt for GPTBot and OAI-SearchBot opt-outs — developers.openai.com/…/bots (verified 2026-08-21)

    Limits

    The mechanism is real; the audit’s original benefit — security and privacy, keeping internal endpoints out of training data — is contradicted by the standard it rests on. RFC 9309 states outright that “The Robots Exclusion Protocol is not a substitute for valid content security measures”, and warns that listing paths makes them publicly discoverable, directing operators to HTTP authentication instead. The old guidance therefore told site owners to publish a map of their admin surface.

    The agent traffic most likely to reach /admin/ or /api/ is also exempt. OpenAI states that for ChatGPT-User, “Because these actions are initiated by a user, robots.txt rules may not apply”. Perplexity states that Perplexity-User “generally ignores robots.txt rules” (docs.perplexity.ai/…/bots, verified 2026-08-21). Finally there is a domain-fit limit that shaped the rewrite: /api/ is precisely the surface an agent wants, so a high-priority failure telling every site to disallow it works against the outcome this project exists to improve.

    All four points are why the audit was rewritten to crawl hygiene over observed URL families, at low priority, rather than kept as a security check.

    How it scores

    Two AI vendors document path-level Disallow for their own AI crawlers with literal directory examples, on top of a ratified standard whose matching rule they implement. That is documented consumer behaviour for exactly the signal this audit inspects.

    Sources