Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/no-nofollow

    Page-level nofollow directives

    What it checks

    Reads page-level nofollow in robots metadata and X-Robots-Tag headers on each scanned page. Applebot documents that this directive stops link traversal on that page. The audit does not inspect individual anchors or prove that linked pages cannot be discovered through other sources.

    Why it matters

    A page-level <meta name="robots" content="nofollow"> (or ‘none’) stops documented AI-serving crawlers from following any link on that page; per-link rel="nofollow" is treated by Google as a hint that links ‘will generally not be followed’. Applying either to internal navigation therefore reduces discovery of the linked pages. Falsifiable: if pages linked only from nofollowed internal links are crawled and indexed at the same rate as normally linked pages, the claim fails.

    Evidence

    Grade: A — Apple’s Applebot documentation names the behaviour directly. Among the supported robots meta directives it lists “nofollow: Applebot won’t follow any links on the page” and “none: Applebot won’t index, snippet, or follow links on the page”. Both are honoured in the X-Robots-Tag header as well. A named agent, a named directive and a stated consequence is grade A. The grade covers the page-level directive only. Apple says nothing about rel="nofollow" on individual anchors, and Google treats that attribute as a hint whose links “will generally not be followed” — so flagging a single per-link nofollow cannot cite this evidence, and the audit does not.

    Apple’s current Applebot documentation is unambiguous and names the behavior directly: among supported robots meta directives it lists ‘nofollow: Applebot won’t follow any links on the page’ and ‘none: Applebot won’t index, snippet, or follow links on the page’. Applebot also supports these via the X-Robots-Tag HTTP header, and falls back to Googlebot’s robots.txt rules when Applebot is not named — so Google-targeted directives leak into Apple’s AI-grounding crawl. Google’s own link-qualification documentation states that links marked nofollow, sponsored or ugc ‘will generally not be followed’.

    Since Google’s AI features require the target page to be ‘indexed and eligible to be shown in Google Search with a snippet’, suppressing traversal to a page suppresses its AI eligibility by the same chain established in signal 1. Two major vendors documenting the consumer behavior in their own crawler docs is what grade A requires.

    Counter-evidence: Two significant qualifications. First, Apple’s documented nofollow is the PAGE-LEVEL meta robots directive, not per-link rel="nofollow" — most audits conflate these, and Apple’s docs say nothing about the rel attribute on individual anchors. An audit that flags a single rel="nofollow" internal link cannot cite the Applebot page as support. Second, Google demoted nofollow from directive to hint in September 2019. It states plainly that ‘the linked pages may be found through other means, such as sitemaps or links from other sites, and thus they may still be crawled’.

    So nofollow does not reliably prevent discovery even for Google. OpenAI, Anthropic and Perplexity documentation is entirely silent on nofollow, with no evidence GPTBot, ClaudeBot or PerplexityBot honors it in either form. Scope the audit to meta robots nofollow/none on indexable pages, and treat per-link rel=nofollow on internal navigation as a weaker informational finding.

    Limits

    The audit reads nofollow only under a name an operator documents reading. It does not recognize none or exempt legitimate utility pages. These limits remain pending. The finding asks for a policy review, not removal of every intentional restriction. Metadata and response headers remain checkable when the page body has no readable text.

    How it scores

    • No scanned pages: notApplicable.
    • No scanned page carries the directive: pass.
    • Some scanned pages carry the directive: warn.
    • Every scanned page carries the directive: fail.

    The sample is the selected pages, not the whole site. Page order does not change the result or the representative affected URL. Evidence lists up to five sorted URLs and the number of additional affected pages. Long evidence text is bounded by the result schema.

    Sources