Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    agent-interfaces/mcp-tool-description-coverage

    Tool Self-Description Coverage

    What it checks

    Deterministic coverage metrics over the tool surface. What fraction of tools carry a description. What fraction of every input parameter, walked recursively through properties, carries a description. What fraction declare an outputSchema and a title, and whether the server ships top-level instructions. No LLM judging — pure presence and length counting against declared thresholds.

    Why it matters

    A tool description and its parameter descriptions are the entire basis on which a model decides whether and how to call it — they are the only prose the model ever sees about the tool. The spec states the documented purpose of outputSchema directly: ‘Guiding clients and LLMs to properly parse and utilize the returned data’, and ‘Enabling strict schema validation of responses’. It defines instructions as ‘natural-language guidance for LLMs on how to use this server effectively’.

    The falsifiable claim is narrow and structural, not aesthetic. A parameter with no description, and no enum, format or pattern, gives the model no way to derive a legal value. It must guess, and guessed values surface as tool-execution errors and retry loops. Coverage is measured, not judged; only the pass thresholds are this project’s convention, which is why this is graded B rather than A.

    Evidence

    • Playwright: Auto-waiting / Actionability checks — Microsoft (vendor-doc, URL verified 2026-08-20)
    • Before click/check/fill/selectOption, Playwright enforces five checks: Visible (non-empty bounding box, not visibility:hidden), Stable (same bounding box over 2 animation frames), Receives Events (element is the hit target at the action point — overlays cause failure), Enabled (not [disabled]/aria-disabled), Editable (not readonly/aria-readonly). Fill requires visible+enabled+editable. This is the exact gate every Playwright-based agent (Playwright-MCP, browser-use, most CUA harnesses) passes through, so each check is a directly testable site-side failure cause.
    • MCP Specification 2026-07-28 — Tools — Model Context Protocol (spec, URL verified 2026-08-20)
    • tools/list result set MUST NOT vary per-connection or as a side effect of other requests (MAY vary by authorization). Servers SHOULD return tools in deterministic order — rationale given verbatim: enables client caching and ‘improves LLM prompt cache hit rates’. inputSchema MUST be a valid JSON Schema object (not null); defaults to JSON Schema 2020-12. Tool names SHOULD be 1-128 chars, case-sensitive, only [A-Za-z0-9_.-], unique within a server. Full x-mcp-header constraint list including static-reachability rule (chain of only properties keys; never through items/oneOf/anyOf/allOf/not/if/then/else/$ref). Clients MUST exclude violating tools from tools/list. If outputSchema present, servers MUST conform. Clients MUST treat annotations as untrusted.

    How it scores

    Tier per evidence policy: scored — grade B meets the A/B bar required for scored audits.

    Example failure

    A booking server exposes create_reservation with a good tool-level description but declares required: ["property_id","rate_plan","guest_count"] where rate_plan is {"type":"string"} with no description, no enum and no pattern. The model has no way to know that legal values are the four internal codes FLEX/NREF/CORP/GRP, so it invents plausible strings like “flexible” and “non-refundable”. Every booking attempt fails validation on the first try; the agent burns two or three retry turns per reservation, and some clients abandon the tool after repeated errors.

    Sources