Machine Discovery
Check the files and links that help machines find your pages. Each check includes its proof, limits and effect on the score.
machine-discovery/agent-commerce-feed-parityGrade AScoredProduct pages carry the fields an agent-commerce feed needs
Samples product pages from the sitemap and audits each against the union of OpenAI's Product Feed Spec and Google Merchant Center's required attributes, using the PDP's structured data as the auditable proxy for feed eligibility. Reports a per-field pass rate plus a separate agent-commerce gap for the fields Google's rich-result validator never asks for, and cross-checks the JSON-LD price against the price the page renders.
machine-discovery/ai-crawler-surface-reachabilityGrade AScoredAI crawlers can reach the discovery surfaces the site advertises
Evaluates robots.txt per named AI user-agent against the exact URLs the site advertises for indexing — the Sitemap: targets, the autodiscovered RSS/Atom/JSON feeds, and a sample of the URLs listed inside the sitemap tree — and flags the self-contradiction of advertising a discovery surface the same file forbids.
machine-discovery/ai-file-deliveryGrade BAdvisoryAI files are delivered correctly
AI agents use the Content-Type header to decide how to parse a file, and caching headers to avoid re-downloading one that has not changed. This audit reports both for every AI file the scan fetched.
machine-discovery/conditional-request-supportGrade BScoredDiscovery surfaces answer conditional requests
Fetches robots.txt, the sitemaps and the feeds twice identically, then once with `If-None-Match` and once with `If-Modified-Since`, and reports what came back. A surface with no validator cannot be revalidated at all; one whose `ETag` changes while its body does not is worse, because every poll looks like a change.
machine-discovery/cors-ai-filesGrade CAdvisoryCORS on AI files
Without CORS headers, agent code running inside a browser origin cannot fetch your llms.txt or AI catalog cross-origin. Server-side crawlers, agent backends and MCP clients are not browsers and are unaffected.
machine-discovery/discovery-index-coverageGrade BScoredPages are covered by a discovery index
Every scanned page should be listed in a discovery index — the sitemap (including its sub-sitemaps) or llms.txt — so AI crawlers can find it without relying on the link graph.
machine-discovery/feed-entry-identity-and-canonical-integrityGrade BScoredFeed entries have stable identities that resolve to their canonical pages
Checks the identity half of a feed: that every entry carries exactly the id and timestamp its format requires, that no id repeats, and that item links are absolute HTTPS URLs which match the `rel="canonical"` of the page they open — no redirect, no tracking parameters the canonical does not carry.
machine-discovery/in-content-linksGrade AScoredIn-content internal links
Contextual links inside the page body — not the nav or footer — are how AI crawlers discover related pages and read the relationships between them. Crawlers that do not execute JavaScript see only the links present as <a href> in the served HTML.
machine-discovery/llms-full-txtGrade CAdvisoryllms-full.txt present
llms-full.txt is a companion convention to llms.txt: the site's content in one markdown file. No AI vendor documents reading it from a website, so this check reports whether the file is served and never scores it.
machine-discovery/llms-txt-existsGrade CAdvisoryllms.txt exists
llms.txt is a community convention: a markdown index of your site at /llms.txt. No AI vendor documents a crawler or agent that reads it, and Google states Search ignores it, so this check is reported and never scored. Chrome Lighthouse checks the same three conformance rules and treats a missing file as not applicable.
machine-discovery/llms-txt-link-descriptionsGrade CAdvisoryllms.txt links include descriptions
The llms.txt format allows a short description after each link. Reports how many links carry one. No AI vendor documents reading llms.txt, so this check is reported and never scored.
machine-discovery/llms-txt-links-validGrade CAdvisoryllms.txt links are valid
Reports whether the links inside a published llms.txt resolve. Link validity is spec-optional and no known consumer enforces it — Chrome Lighthouse, the only shipping checker, fetches no link at all — so this check is reported and never scored.
machine-discovery/llms-txt-structureGrade CAdvisoryllms.txt is well-formed
The llms.txt format defines a blockquote summary under the H1 and H2 sections grouping the link lists. Both are optional in the spec, so this check is advisory: it reports the shape of the file, it does not score it.
machine-discovery/no-broken-ai-endpointsGrade AScoredNo broken AI endpoints
AI agents follow URLs in your ai-catalog.json, llms.txt, and navigation.json to build a map of your site's AI-consumable resources. Broken links cause agents to lose trust in your manifest files entirely, potentially ignoring all listed endpoints. Fix or remove broken URLs.
machine-discovery/no-broken-linksGrade AScoredNo broken internal links
Broken internal links create dead ends for AI crawlers and waste their limited crawl budget.
machine-discovery/root-text-file-resolution-integrityGrade BScoredThe origin serves and correctly 404s root-level .txt resources
Fetches two root-level `.txt` files with random names that cannot exist. Both must answer 404 or 410. An origin that answers 200 instead has a catch-all, which makes every probe-based discovery file — `llms.txt`, `ai.txt`, `security.txt`, the IndexNow key file — impossible to tell apart from a soft 404. Also checks that `/robots.txt` is served as `text/plain`.
machine-discovery/rss-feedGrade BScoredRSS/Atom feed link present
RSS/Atom feeds let AI agents track new and updated content without re-crawling your entire site. The <head> autodiscovery link is reported alongside the feed, not scored on its own.
machine-discovery/rss-feed-contentGrade CAdvisoryRSS feed content complete
Reports how many feed items carry full article text (more than 500 characters). Google and the Common Crawl news crawler take URLs from feeds and fetch the page itself, no AI vendor documents reading feeds, and excerpt feeds are valid, so this check is reported and never scored.
machine-discovery/sitemap-absolute-urlsGrade BScoredSitemap uses absolute URLs
Sitemap URLs must be absolute (starting with https://) so AI crawlers can resolve them without ambiguity.
machine-discovery/sitemap-existsGrade AScoredsitemap.xml exists
AI crawlers use your sitemap to discover all pages without following links. Without it, pages may never be indexed by AI search engines.
machine-discovery/sitemap-lastmodGrade AScoredSitemap has lastmod dates
AI crawlers use <lastmod> to decide which pages to re-index and which to skip. Without these dates, crawlers must re-fetch every page on every visit.
machine-discovery/sitemap-lastmod-verifiabilityGrade AScoredSitemap lastmod values are verifiable against the pages
Cross-validates sampled sitemap <lastmod> values against three independent page-level modification signals — the Last-Modified response header, JSON-LD dateModified/datePublished, and article:modified_time — and scores agreement rather than presence. Catches the two dominant failure modes when the page dates disagree: the build stamp (every URL updated on every deploy) and the frozen value (the CMS never updates it). A deploy-time Last-Modified header never vouches for a recent run of identical stamps.
machine-discovery/three-way-freshness-lagGrade BScoredThe sitemap and the feed are as fresh as the site itself
Compares the newest date the pages themselves publish against the newest `<lastmod>` in the sitemap and the newest entry in the feed. A surface that trails the site by more than a week is regenerated on a slower cadence than publication, so a crawler polling it sees a site that stopped publishing. Also checks that a feed’s own build timestamp is not older than its newest item, and that its items are in newest-first order.
machine-discovery/websub-hub-advertisementGrade BScoredFeeds that advertise a WebSub hub carry exactly one absolute self link
Reads the WebSub discovery links on each feed — the `Link:` response headers first, as the specification requires, then the document — and, on each feed that advertises a hub, checks the shape the W3C Recommendation asks for: exactly one absolute `rel=self`, and every `rel=hub` an absolute URL whose server answers a HEAD. HTTP hubs are allowed, and a hub that refuses the scanner (401, 403, 429) counts as present. The self URL may differ from the URL the feed was fetched from; the Recommendation allows it. A feed with no hub is not judged. Google Search documents receiving WebSub notifications from Atom and RSS feeds; no LLM vendor documents subscribing.