Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    agent-interfaces/mcp-tool-description-coverage

    Tool Self-Description Coverage

    What it checks

    Deterministic coverage metrics over the tool surface. What fraction of tools carry a description. What fraction of every input parameter, walked recursively through properties, carries a description. What fraction declare an outputSchema and a title, and whether the server ships top-level instructions. No LLM judging — pure presence and length counting against declared thresholds.

    Why it matters

    A tool description and its parameter descriptions are the entire basis on which a model decides whether and how to call it — they are the only prose the model ever sees about the tool. The spec states the documented purpose of outputSchema directly: ‘Guiding clients and LLMs to properly parse and utilize the returned data’, and ‘Enabling strict schema validation of responses’. It defines instructions as ‘natural-language guidance for LLMs on how to use this server effectively’.

    The falsifiable claim is narrow and structural, not aesthetic. A parameter with no description, and no enum, format or pattern, gives the model no way to derive a legal value. It must guess, and guessed values surface as tool-execution errors and retry loops. Coverage is measured, not judged; only the pass thresholds are this project’s convention, which is why this is graded B rather than A.

    Evidence

    • MCP Specification 2026-07-28 — Tools — Model Context Protocol (spec, verified 2026-10-17)
    • A tool carries a description, an optional title, an inputSchema and an optional outputSchema. Tools are “model-controlled”: the model selects and invokes them from what tools/list returns. The spec’s own examples give every input property a description. It states that an output schema helps “clients and LLMs” by “Guiding clients and LLMs to properly parse and utilize the returned data” and “Enabling strict schema validation of responses”.
    • MCP Specification 2026-07-28 — Discovery — Model Context Protocol (spec, verified 2026-10-17)
    • DiscoverResult.instructions is “Optional natural-language guidance for LLMs on how to use this server effectively”.
    • Anthropic — define tools — Anthropic (vendor-doc, verified 2026-10-17)
    • A documented consumer of tool descriptions. Claude’s tool-use system prompt is built from the tool definitions. The best-practice list says: “Provide extremely detailed descriptions. This is by far the most important factor in tool performance.” A good description covers “What each parameter means and how it affects the tool’s behavior”. The page’s poor example is a tool whose parameter has no description.

    Grade: B — the MCP spec defines every field the audit counts, and Anthropic documents that descriptions drive tool performance. The pass thresholds are this project’s convention, not a source’s, so the grade is B rather than A.

    How it scores

    Tier per evidence policy: scored — grade B meets the A/B bar required for scored audits.

    Example finding

    A booking server exposes create_reservation with a good tool-level description but declares required: ["property_id","rate_plan","guest_count"] where rate_plan is {"type":"string"} with no description, no enum and no pattern. The model has no way to know that legal values are the four internal codes FLEX/NREF/CORP/GRP, so it invents plausible strings like “flexible” and “non-refundable”. Every booking attempt fails validation on the first try; the agent burns two or three retry turns per reservation, and some clients abandon the tool after repeated errors.

    Sources