Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/meta-external-agent

    Meta-ExternalAgent allowed

    What it checks

    Meta-ExternalAgent collects pages for Meta’s foundation-model training and for indexing content directly into Meta products, and Meta documents it as respecting robots.txt.

    This reads the robots.txt rules that actually apply to the token — its own group if it has one, otherwise the catch-all — and reports whether they let it fetch the site root. A named group is not required: under RFC 9309 §2.2.1 an open catch-all grants a named crawler the same access a named group would.

    Why it matters

    Disallowing meta-externalagent stops Meta collecting the site for foundation-model training and direct product indexing, and Meta states the agent respects robots.txt. Disallowing meta-externalfetcher does not reliably stop fetches, because Meta reserves a user-request exemption.

    Evidence

    Meta-ExternalAgent allow/block state in robots.txt

    Meta’s web crawlers page documents meta-externalagent as crawling ‘for use cases such as training foundation AI models or improving products by indexing content directly’, with no stated robots.txt exemption. The newer Meta-WebIndexer ‘navigates the web to improve Meta AI search result quality for users’ and helps ‘cite and link to your content in Meta AI’s responses’. That makes Meta-WebIndexer the allow-side visibility token and meta-externalagent the training-side block token. Cloudflare Radar confirms Meta-ExternalAgent among the top five AI crawlers overall and at 13.9% share in the Computer & Electronics vertical (Aug 2025), so it is documented ACTIVE at scale.

    Limits

    Two documented robots.txt exemptions in the same family must not be conflated with meta-externalagent. meta-externalfetcher ‘fetches individual links at a user’s request’, and ‘may bypass robots.txt rules’. facebookexternalhit may bypass robots.txt for ‘security or integrity checks, such as checking for malware or malicious content’. A meta-externalfetcher disallow should therefore be reported informatively, not scored as an effective control.

    How it scores

    Meta’s web-crawlers page documents meta-externalagent as crawling “for use cases such as training foundation AI models or improving products by indexing content directly”, with no stated robots.txt exemption, and gives a literal per-directory example. A vendor naming its own token, its purpose and the directive that governs it is the grade-A bar. The grade attaches to this token specifically: the same page reserves a user-request exemption for meta-externalfetcher, which “may bypass robots.txt rules”, and a security exemption for facebookexternalhit. Conflating the three would spend an A on two agents that do not honour the directive.

    Sources