Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    access-crawl-control/mistralai-user

    MistralAI-User allowed

    What it checks

    Without an explicit robots.txt rule, MistralAI-User may still crawl your site but has no signal that it is welcome. Adding an explicit allow rule improves your visibility in AI-powered search and ensures consistent crawler behavior.

    Why it matters

    Mistral publishes three separate tokens with distinct consequences:

    Evidence

    MistralAI-User allow/block state in robots.txt

    • a MistralAI-Training disallow blocks training-corpus collection;
    • a MistralAI-Index disallow removes the site from Mistral search, and therefore from Vibe answers;
    • MistralAI-User governs which sites user-initiated Vibe requests may access.

    Unusually, Mistral asserts no user-initiated robots.txt exemption.

    Mistral’s robots page documents all three UAs, each carrying ‘+https://docs.mistral.ai/robots’. MistralAI-User/1.0. ‘When users ask Vibe a question, it may visit a web page to help answer’. robots.txt ‘governs which sites user requests can access’. The agent is ‘not used for crawling the web in any automatic fashion, nor to crawl content for generative AI training’. MistralAI-Index/1.0: ‘It indexes content for Mistral search, which helps answer user questions in Vibe’, content ‘not used for generative AI training of any kind’. MistralAI-Training/1.0: ‘Webmasters can disallow this user agent in their robots.txt file.’ The clean training/index/user separation makes per-token scoring straightforward and MistralAI-User is notably the only major user-initiated agent whose vendor does not claim a robots.txt exemption.

    Limits

    Mistral is the most extractive operator by 2026 crawl-to-refer measurement (reported at ~3,389 pages crawled per referral sent, worse than Anthropic and far worse than OpenAI), so the allow-side referral argument for MistralAI-Index is weak. Mistral bots do not appear in Cloudflare Radar’s Aug 2025 named top-five breakdowns, so historical volume was small. Audits keyed only to ‘MistralAI-User’ will miss the two higher-impact tokens.

    How it scores

    Mistral publishes all three tokens on one page, with distinct stated consequences. For this one it says the thing that matters: robots.txt “governs which sites user requests can access”, and the agent is “not used for crawling the web in any automatic fashion”. Like Anthropic and unlike OpenAI, Perplexity and Meta, Mistral asserts no user-initiated exemption — so the directive is expected to be honoured. Named agent, named directive, stated behaviour: grade A. The referral case for the sibling index token is weak, at a reported ~3,389 pages crawled per referral sent, but that is an argument about worth rather than about evidence.

    Sources