Content Extraction
Check whether machines can read the main content of your pages. Each check includes its proof, limits and effect on the score.
content-extraction/article-elementGrade AScored<article> used for content
RAG systems chunk content by <article> boundaries for vector embedding, treating each article as an independent retrieval unit. Without <article> tags, AI chunking algorithms fall back to arbitrary text splitting, which fragments related content across multiple embeddings and reduces answer quality.
content-extraction/aside-elementGrade BScored<aside> for supplementary content
Mozilla Readability and trafilatura delete <aside> subtrees before extraction, and Chromium exposes <aside> as a complementary landmark in the accessibility tree agents read. Wrapping sidebars, promos and related-links blocks in <aside> is what keeps them out of the text an LLM ingests; a sidebar left in a bare <div> is extracted as if it were article body.
content-extraction/boilerplate-taxGrade BScoredBoilerplate tax across the crawl (unique tokens per fetch)
Samples pages across the crawl, finds the five-word windows that appear on at least 80% of them, and reports how many of the tokens an agent pays for are distinct information rather than repeated chrome. Site-level rather than page-level: the cost of boilerplate is only visible across fetches.
content-extraction/code-languageGrade CAdvisoryCode blocks have language annotations
AI agents use language annotations on code blocks to apply the correct syntax understanding and provide accurate code explanations. Without them, agents must guess the programming language, which can lead to incorrect interpretations in AI-generated code answers.
content-extraction/content-depthGrade BScoredSufficient content depth
AI RAG systems need sufficient content depth to generate accurate, detailed answers. Pages with fewer than 300 words provide too little context for meaningful vector embeddings, causing your content to rank poorly in retrieval and be excluded from AI-generated responses.
content-extraction/css-hidden-ghost-contentGrade AScoredGhost content: CSS-hidden text ingested as visible
Find text that is hidden from human readers by an external stylesheet class but is invisible-as-hidden to every extractor an agent uses, and size it in tokens. Fail if class-hidden text exceeds 15% of the page's total text tokens or 1,000 tokens absolute; separately fail on near-duplicate hidden blocks (a mobile nav or tab-panel set duplicating visible content). Report contradiction risk when hidden text contains prices, availability, or dated claims.
content-extraction/data-tablesGrade BScoredData tables properly structured
AI agents use <thead> and <th> elements to understand column headers and interpret table data correctly. Without proper structure, agents cannot map cell values to their column meanings, leading to garbled data extraction in AI-generated comparisons and summaries.
content-extraction/extraction-determinismGrade BScoredExtraction determinism (multi-extractor agreement)
Runs three independent main-content extractors over the page — `@mozilla/readability`, a semantic-container selector, and a text-density scorer — and compares their output pairwise with five-word shingles. Where they disagree, what an agent quotes from this page is decided by its pipeline rather than by the page.
content-extraction/fake-headingsGrade BScoredNo fake headings
AI agents chunk and outline page content by reading real <h1>–<h6> tags. When a page styles a <div>, <span>, <p>, or <b> to look like a heading (large text, bold weight, "heading" classes) instead of using a semantic heading element, that text is invisible to the agent's document outline — sections cannot be navigated, summarized, or cited correctly. This audit is distinct from the sequential-heading check (content-extraction/sequential-headings), which verifies that real headings appear in the right order, while this audit catches content that impersonates headings without using heading tags at all. Replace styled generic elements with the appropriate <h1>–<h6> level.
content-extraction/figure-figcaptionGrade CAdvisory<figure> + <figcaption> usage
AI agents use <figcaption> to understand the purpose and context of figures beyond what alt text provides. Without captions, agents treat figures as opaque image containers with no semantic meaning, missing opportunities to cite your visual data in AI-generated answers.
content-extraction/header-footerGrade AScored<header> and <footer> landmarks
AI agents use <header> and <footer> landmarks to identify and exclude boilerplate content (navigation, copyright, links) from primary content extraction. Without these landmarks, agents may include footer disclaimers or nav menus in their content summaries.
content-extraction/hydration-payload-shareGrade AScoredInlined hydration-state payload share
Detect and size serialized framework state inlined in the HTML document: <script id="__NEXT_DATA__">, self.__next_f.push( flight chunks, window.__NUXT__, __remixContext, window.__APOLLO_STATE__, window.__INITIAL_STATE__, <script type="application/json"> islands, and Astro/Svelte island props. Three independent failure conditions: (1) any single state payload > 128 kB, (2) total state payload > 30% of document tokens, (3) state payload duplicates > 50% of the main-content text (content shipped twice in one response).
content-extraction/image-alt-textGrade AScoredImage text-alternative coverage
An image with no text alternative has no accessible name, so it is an unnamed node in the accessibility-tree snapshots agent toolkits send to a model — Playwright MCP, Claude-in-Chrome read_page, Chrome DevTools take_snapshot — and it carries no subject matter for Google Images, which states it uses alt text to understand what an image shows. A multimodal agent that fetches the image bytes can caption it without one; a text-only crawler or a snapshot-driven agent cannot.
content-extraction/json-ld-duplication-massGrade CAdvisoryJSON-LD duplication mass
Counts what the page's JSON-LD costs in `o200k_base` tokens, finds nodes declared twice across blocks, and measures how much of any long prose property — `articleBody`, `description`, `reviewBody` — repeats text already present in the DOM. Reported, not scored: duplication is a cost an operator may have chosen, and no consumer path proves it changes an answer.
content-extraction/language-attributeGrade AScoredLanguage attribute
AI agents use the lang attribute to select the correct language model and tokenizer when processing your content. Without it, agents may misinterpret content language, leading to poor translations or incorrect answers in multilingual AI systems.
content-extraction/main-elementGrade AScored<main> element present
AI scrapers use <main> to identify primary content and discard nav/footer chrome, reducing hallucination risk from boilerplate text. Without <main>, agents must guess which content is primary versus navigational, often ingesting menus and footers into their context window.
content-extraction/markdown-alternateGrade AScoredMarkdown alternate: resolvable, faithful, cheaper
Where a site serves a markdown alternate of a page — by a declared `alternate` link, by `url + ".md"`, or by `Accept: text/markdown` — checks that it resolves as `text/markdown`, still carries the page's headings and prose, and costs fewer tokens than the HTML. A site that serves no markdown alternate at all is reported as not applicable: the documented consumers are interactive coding agents, and no cited source measures a cost to a site that serves none.
content-extraction/preamble-taxGrade BScoredPreamble tax: tokens before the first content token
Measures how many `o200k_base` tokens an agent must stream past before the first sentence of the main content appears, by locating the extracted content inside the raw response body. Reports the offset in tokens and as a share of the document, and names the single heaviest block sitting in front of the content.
content-extraction/section-headingsGrade BScored<section> elements have headings or labels
AI agents use section headings to build a topic map of your page for retrieval-augmented generation (RAG). Unlabeled sections are opaque to AI systems that chunk content by semantic boundaries, reducing the quality of retrieved context for answer generation.
content-extraction/semantic-listsGrade BScoredSemantic list usage
AI agents recognize <ul>, <ol>, and <dl> as structured data lists and extract them as bullet points, numbered steps and term/definition pairs in generated answers. Content formatted as styled divs or manually numbered paragraphs is invisible to list-extraction algorithms, so your feature lists, how-to steps and glossary entries will not be surfaced as structured answers.
content-extraction/sequential-headingsGrade BScoredSequential heading hierarchy
AI systems build content outlines from headings to understand document structure. Skipped levels (e.g., h1 to h3 without h2) break the hierarchy, causing agents to misinterpret section nesting and produce inaccurate content summaries. Fix heading levels to follow a sequential order.
content-extraction/server-renderedGrade BScoredServer-rendered content
AI crawlers like GPTBot and ClaudeBot do not execute JavaScript. Content only visible after JS execution is completely invisible to them, meaning your site effectively has no content in AI knowledge bases. Use SSR (server-side rendering) or SSG (static site generation) to serve content in the initial HTML response.
content-extraction/server-responsivenessGrade BScoredServer responsiveness
AI crawlers fetch fewer pages per session from a slow origin, and a user-triggered agent fetch that outlasts the client budget is abandoned before bytes arrive. This measures the median time to first byte across the crawled pages, not a single cold sample.
content-extraction/single-h1Grade BScoredSingle h1 per page
AI agents use the single <h1> as the authoritative title of the page for content indexing and answer generation. Ensure exactly one <h1> per page.
content-extraction/svg-bloatGrade BScoredSVGs not bloating agent context
When an LLM converts your HTML to Markdown or reads raw markup, every inline SVG is inlined as thousands of path-data tokens, and every base64 data URI is inlined verbatim. Decorative icon sprites, charts, and complex illustrations can silently consume tens of thousands of tokens of agent context per page — "SVG context poisoning" — crowding out the actual content the agent should read. SVGs marked aria-hidden="true" or role="presentation" are stripped by most accessibility-tree extractors and do not count. Keep visible SVGs small, move decorative ones behind aria-hidden, and prefer raster images or CSS for complex graphics. Inlined base64 assets are priced in real `o200k_base` tokens and reported separately, because their fix differs: move the asset to a real URL with descriptive alt text.
content-extraction/time-elementGrade CAdvisory<time datetime=""> used for dates
AI agents use <time datetime> elements to reliably parse dates for freshness scoring and temporal reasoning. Without machine-readable dates, agents must regex-parse human-readable date formats, which frequently fails across locales and ambiguous formats like "01/02/2025".
content-extraction/token-ratioGrade BScoredLean token-to-content ratio
AI agents pay for every token of raw HTML they download, but only the main content carries meaning. This audit counts both sides with a real BPE tokenizer (`o200k_base`): the numerator is the text `@mozilla/readability` extracts — the extractor most of the industry deploys — and the denominator is the whole delivered document. The result is a signal-density index, and the report breaks the denominator into script, style, comment, content and structure tokens so the finding names the bucket to attack. A ratio under 5% means an agent parses 20 tokens of noise for every token of content; under 15% still wastes most of the context window on boilerplate.