Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/applebot-extended

    Applebot-Extended allowed

    What it checks

    Without an explicit robots.txt rule, Applebot-Extended may still crawl your site but has no signal that it is welcome. Adding an explicit allow rule improves your visibility in AI-powered search and ensures consistent crawler behavior.

    Why it matters

    Applebot-Extended is a robots.txt-only token: disallowing it opts the site’s already-crawled content out of training Apple’s foundation models (Apple Intelligence) without removing the site from Siri, Spotlight or Safari search results. Allowing Applebot while disallowing Applebot-Extended is the documented way to keep search presence and refuse training.

    Evidence

    Applebot-Extended allow/block state in robots.txt

    Apple maintains two dedicated support pages for this: ‘About Applebot’ (support.apple.com/en-us/119829) and ‘Applebot model training and individual privacy rights’ (support.apple.com/en-us/120320). Apple’s stated position is that web publishers can use standard robots.txt directives either to stop Applebot crawling, or separately to direct Apple not to use their content to train Apple’s foundation models. The two tokens are independent — an Applebot allow does not imply an Applebot-Extended allow. Because it is not a fetching crawler, no traffic-share data exists or should be expected.

    Limits

    One caveat on this verification: both Apple support pages are client-side rendered, and the automated fetcher recovered only the document titles, not the body text. The URLs resolve and are unambiguously Apple’s canonical Applebot documentation, but Apple’s exact sentences could not be quoted; the specifics above are corroborated by secondary sources rather than by direct extraction. The wording should be confirmed before any of it is published as a verbatim quote. Additionally, because Applebot-Extended emits no requests, its effect is entirely unobservable from server logs — publishers cannot verify Apple honors it.

    How it scores

    Apple maintains two dedicated support pages for this token and documents the split it enables: disallowing Applebot-Extended opts already-crawled content out of foundation-model training while leaving Applebot, and therefore Siri, Spotlight and Safari search presence, untouched. A vendor documenting its own token and the exact effect of the directive is the grade-A bar. The one honesty caveat is on the verification, not the claim: both Apple pages are client-side rendered, and the fetcher recovered titles rather than body text, so the sentences are corroborated rather than quoted.

    Sources