Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/meta-external-fetcher

    Meta-ExternalFetcher allowed

    What it checks

    Without an explicit robots.txt rule, Meta-ExternalFetcher may still crawl your site but has no signal that it is welcome. Adding an explicit allow rule improves your visibility in AI-powered search and ensures consistent crawler behavior.

    Why it matters

    Disallowing meta-externalagent stops Meta collecting the site for foundation-model training and direct product indexing, and Meta states the agent respects robots.txt. Disallowing meta-externalfetcher does not reliably stop fetches, because Meta reserves a user-request exemption.

    Evidence

    Meta-ExternalAgent allow/block state in robots.txt

    Meta’s web crawlers page documents meta-externalagent as crawling ‘for use cases such as training foundation AI models or improving products by indexing content directly’, with no stated robots.txt exemption. The newer Meta-WebIndexer ‘navigates the web to improve Meta AI search result quality for users’ and helps ‘cite and link to your content in Meta AI’s responses’. That makes Meta-WebIndexer the allow-side visibility token and meta-externalagent the training-side block token. Cloudflare Radar confirms Meta-ExternalAgent among the top five AI crawlers overall and at 13.9% share in the Computer & Electronics vertical (Aug 2025), so it is documented ACTIVE at scale.

    Limits

    Two documented robots.txt exemptions in the same family must not be conflated with meta-externalagent. meta-externalfetcher ‘fetches individual links at a user’s request’, and ‘may bypass robots.txt rules’. facebookexternalhit may bypass robots.txt for ‘security or integrity checks, such as checking for malware or malicious content’. A meta-externalfetcher disallow should therefore be reported informatively, not scored as an effective control.

    How it scores

    The grade-A material in this signal belongs to meta-externalagent, whose robots.txt compliance Meta documents without exemption. It transfers to this audit only as far as the token being real and documented: Meta states that meta-externalfetcher “fetches individual links at a user’s request” and “may bypass robots.txt rules”. A directive the vendor says its agent may ignore cannot carry a pass or a failure, which is why this audit reports the declaration’s presence and its stated unreliability rather than scoring compliance.

    Sources