Opens in a new tabSkip to content
Agent LighthouseAgent Lighthouse

    Searches the text of every published page. The evidence sources themselves are not in this index — search all of them on the trusted sources page.

    GitHub ↗
    Browse checks and page contents
    content-extraction/token-ratio

    Lean token-to-content ratio

    What it checks

    AI agents pay for every token of raw HTML they download, but only the visible text carries meaning. This audit compares the character weight of the raw HTML against the extracted main-content text to produce a “context bloat score”: the share of the page that is actual content rather than markup, scripts, and styles. The ratio is evaluated on the homepage (the first crawled page), which is the entry point agents most often fetch. A ratio under 5% means an agent parses 20 tokens of noise for every token of content; under 15% still wastes most of the context window on boilerplate. Unlike content depth (which measures text volume), this measures how efficiently that text is packaged.

    Why it matters

    Content depth and text-to-boilerplate token ratio — Chrome is everything that is not main content: nav, repeated headers and footers, wrapper divs, inline scripts. The larger its share of a page’s serialized bytes and tokens, the smaller the fraction of the page that survives extraction into the model’s context — and the more likely a fixed truncation cap severs real content. Conversely a page with too little actual content relative to its scaffolding gives an extractor nothing substantive to return.

    Inline SVG and DOM bloat consuming LLM context — Deeply nested DOM and large inline SVG inflate the serialized page representation an agent receives, pushing it against fixed truncation caps and depth limits so content below the cut is never seen. The general DOM-size claim is well supported; the SVG-specific claim is that inline path data is pure token cost with no semantic payload, since it carries no accessible name and contributes nothing an LLM can reason about.

    Evidence

    Content depth and text-to-boilerplate token ratio

    Quantified from three independent directions. Cloudflare measured a real blog post at 16,180 HTML tokens against 3,150 in markdown — an 80% reduction. It attributed the delta explicitly to ‘the <div> wrappers, nav bars, and script tags that pad every real web page and have zero semantic value’. The response even ships x-original-tokens and x-markdown-tokens headers, so agents can compute the ratio [cloudflare-markdown-for-agents]. The 2026 observation study measured HTML at about 56,653 input tokens per agent step, against about 6,720 for the accessibility tree.

    That is a gap of roughly 8.4x [observation-reduction-paper]. trafilatura’s stated purpose is to ‘remove the noise consisting of recurring elements (headers and footers, ads, links/blogroll)’ [trafilatura-corefunctions]. Readability does the same job, via link-density and text-density scoring [mozilla-readability-source]. Truncation is real and first-party: Anthropic’s read_page caps output at 50,000 characters and truncates at a line boundary [anthropic-browser-use-tool].

    Inline SVG and DOM bloat consuming LLM context

    Truncation is documented first-party: Anthropic’s read_page caps output at 50,000 characters, truncates at a line boundary, and offers depth (default 15) and ref-scoping as the remedy — an explicit admission that page size forces partial reads [anthropic-browser-use-tool]. The scale of the problem is measured: ‘Some real world DOMs surpass the size of a megabyte’, roughly 1e6 tokens, against 1e3 to 1e4 after downsampling. The D2Snap ablation found DOM hierarchy ‘the strongest among those features’ for LLM performance. Its attribute filter preserves alt, href and aria-*, and discards the rest [dom-downsampling-paper]. Token magnitudes corroborated at ~56,653 HTML tokens per step [observation-reduction-paper] and by Cloudflare’s 80% markdown reduction attributed to semantically empty wrappers and scripts [cloudflare-markdown-for-agents].

    Limits

    Content depth and text-to-boilerplate token ratio — Do not treat ‘less boilerplate is always better’ as proven. The same study that quantifies the token gap found high-capability models perform better on the fuller HTML observation: Claude Sonnet 4.6 by 14.6pp, GPT-5.1 by 17.5pp. They exploit layout information for action grounding. Only weaker models degrade under long inputs [observation-reduction-paper]. No published source defines an acceptable text-to-boilerplate threshold; any specific number an audit uses (e.g. ‘main content must be >40% of tokens’) is invented and must be presented as a heuristic, not as a standard.

    Google states no special optimizations are needed for AI features [google-ai-features-docs]. Word-count style ‘content depth’ minimums in particular have no support in any source found for this domain — score the RATIO with a documented mechanism, not an arbitrary length floor.

    Inline SVG and DOM bloat consuming LLM context — The SVG-specific half is materially weaker than the DOM-size half and should be graded C on its own. No vendor doc, spec or study located for this dossier singles out inline SVG as an agent problem. Mechanically the cost is asymmetric. An inline <svg> without a title or aria-label collapses to a single unnamed node in the accessibility tree, or is omitted from it.

    Its bloat therefore lands on raw-HTML and markdown consumers, not on the a11y-tree agents that dominate this domain. An SVG-bloat audit is really a payload-weight audit, not an agent-perception audit. And bigger is not uniformly worse: strong models gained double-digit points from the larger HTML observation [observation-reduction-paper]. Recommend scoring total serialized DOM size / node depth with the truncation cap as the documented anchor, and demoting the SVG-specific rule to an informative sub-check unless the SVG is also unnamed where it acts as a control.

    How it scores

    Content depth and text-to-boilerplate token ratio — Quantified from three independent directions rather than asserted. Cloudflare measured one real blog post at 16,180 HTML tokens against 3,150 in markdown, an 80% reduction. It attributed the delta to “the <div> wrappers, nav bars, and script tags that pad every real web page and have zero semantic value”. Strong empirical evidence of an effect, with no vendor stating a requirement, is grade B. The grade deliberately does not carry the stronger claim that less boilerplate is always better. The same study found high-capability models performed better on the fuller HTML: Claude Sonnet 4.6 by 14.6pp, GPT-5.1 by 17.5pp. They exploit layout for action grounding.

    Inline SVG and DOM bloat consuming LLM context — The truncation half is documented first-party. Anthropic’s read_page caps output at 50,000 characters, and truncates at a line boundary. It offers a depth limit, default 15, and ref-scoping as the remedy. That is an explicit admission that page size forces partial reads. Real DOMs are reported to exceed model context windows outright. That is a documented consumer with a stated limit but no measured effect on answer quality, which is grade B.

    The SVG-specific half is weaker, and the audit treats it that way. No vendor document, specification or study singles out inline SVG. Mechanically, an <svg> with no title and no aria-label collapses to one unnamed node in the accessibility tree. Its path data therefore costs nothing in a tree-based snapshot, and costs a great deal only in raw-HTML pipelines.

    Sources