Machine Discovery
Check the files and links that help machines find your pages. Each check includes its proof, limits and effect on the score.
machine-discovery/agent-commerce-feed-parityGrade AScoredProduct pages carry the fields an agent-commerce feed needs
Samples product pages from the sitemap and audits each against the union of OpenAI's Product Feed Spec and Google Merchant Center's required attributes, using the PDP's structured data as the auditable proxy for feed eligibility. Reports a per-field pass rate plus a separate agent-commerce gap for the fields Google's rich-result validator never asks for, and cross-checks the JSON-LD price against the price the page renders.
machine-discovery/ai-crawler-surface-reachabilityGrade AScoredAI crawlers can reach the discovery surfaces the site advertises
Evaluates robots.txt per named AI user-agent against the exact URLs the site advertises for indexing — the Sitemap: targets, the autodiscovered RSS/Atom/JSON feeds, and a sample of the URLs listed inside the sitemap tree — and flags the self-contradiction of advertising a discovery surface the same file forbids.
machine-discovery/ai-file-deliveryGrade BAdvisoryAI files are delivered correctly
AI agents use the Content-Type header to decide how to parse a file, and caching headers to avoid re-downloading one that has not changed. This audit reports both for every AI file the scan fetched.
machine-discovery/conditional-request-supportGrade BScoredDiscovery surfaces answer conditional requests
Fetches robots.txt, the sitemaps and the feeds twice identically, then once with `If-None-Match` and once with `If-Modified-Since`, and reports what came back. A surface with no validator cannot be revalidated at all; one whose `ETag` changes while its body does not is worse, because every poll looks like a change.
machine-discovery/cors-ai-filesGrade CAdvisoryCORS on AI files
Without CORS headers, AI agents running in browser contexts cannot fetch your llms.txt or API spec. Browser-based AI tools, ChatGPT plugins, and MCP clients are all blocked by same-origin policy, making your AI-facing files completely inaccessible to cross-origin agents.
machine-discovery/discovery-index-coverageGrade BScoredPages are covered by a discovery index
Every scanned page should be listed in a discovery index — the sitemap (including its sub-sitemaps) or llms.txt — so AI crawlers can find it without relying on the link graph.
machine-discovery/feed-entry-identity-and-canonical-integrityGrade BScoredFeed entries have stable identities that resolve to their canonical pages
Checks the identity half of a feed: that every entry carries exactly the id and timestamp its format requires, that no id repeats, and that item links are absolute HTTPS URLs which match the `rel="canonical"` of the page they open — no redirect, no tracking parameters the canonical does not carry.
machine-discovery/in-content-linksGrade AScoredIn-content internal links
Contextual links inside the page body — not the nav or footer — are how AI crawlers discover related pages and read the relationships between them. Crawlers that do not execute JavaScript see only the links present as <a href> in the served HTML.
machine-discovery/llms-full-txtGrade CAdvisoryllms-full.txt present
llms-full.txt provides the complete content of your site in a single file, allowing AI agents to ingest everything in one request instead of crawling page by page.
machine-discovery/llms-txt-existsGrade CAdvisoryllms.txt exists
llms.txt is a community convention: a markdown index of your site at /llms.txt. No AI vendor documents a crawler or agent that reads it, and Google states Search ignores it, so this check is reported and never scored. Chrome Lighthouse checks the same three conformance rules and treats a missing file as not applicable.
machine-discovery/llms-txt-link-descriptionsGrade CAdvisoryllms.txt links include descriptions
Link descriptions help AI agents understand what each page covers without visiting it, reducing unnecessary crawling.
machine-discovery/llms-txt-links-validGrade CAdvisoryllms.txt links are valid
Reports whether the links inside a published llms.txt resolve. Link validity is spec-optional and no known consumer enforces it — Chrome Lighthouse, the only shipping checker, fetches no link at all — so this check is reported and never scored.
machine-discovery/llms-txt-structureGrade CAdvisoryllms.txt is well-formed
The llms.txt format defines a blockquote summary under the H1 and H2 sections grouping the link lists. Both are optional in the spec, so this check is advisory: it reports the shape of the file, it does not score it.
machine-discovery/no-broken-ai-endpointsGrade AScoredNo broken AI endpoints
AI agents follow URLs in your ai-catalog.json, llms.txt, and navigation.json to build a map of your site's AI-consumable resources. Broken links cause agents to lose trust in your manifest files entirely, potentially ignoring all listed endpoints. Fix or remove broken URLs.
machine-discovery/no-broken-linksGrade AScoredNo broken internal links
Broken internal links create dead ends for AI crawlers and waste their limited crawl budget.
machine-discovery/root-text-file-resolution-integrityGrade BScoredThe origin serves and correctly 404s root-level .txt resources
Fetches two root-level `.txt` files with random names that cannot exist. Both must answer 404 or 410. An origin that answers 200 instead has a catch-all, which makes every probe-based discovery file — `llms.txt`, `ai.txt`, `security.txt`, the IndexNow key file — impossible to tell apart from a soft 404. Also checks that `/robots.txt` is served as `text/plain`.
machine-discovery/rss-feedGrade BScoredRSS/Atom feed link present
RSS/Atom feeds let AI agents track new and updated content without re-crawling your entire site. The <head> autodiscovery link is reported alongside the feed, not scored on its own.
machine-discovery/rss-feed-contentGrade CAdvisoryRSS feed content complete
Full-content feeds allow AI agents to index your articles without visiting each page, reducing crawl load and improving content quality in AI responses.
machine-discovery/sitemap-absolute-urlsGrade BScoredSitemap uses absolute URLs
Sitemap URLs must be absolute (starting with https://) so AI crawlers can resolve them without ambiguity.
machine-discovery/sitemap-existsGrade AScoredsitemap.xml exists
AI crawlers use your sitemap to discover all pages without following links. Without it, pages may never be indexed by AI search engines.
machine-discovery/sitemap-lastmodGrade AScoredSitemap has lastmod dates
AI crawlers use <lastmod> to decide which pages to re-index and which to skip. Without these dates, crawlers must re-fetch every page on every visit.
machine-discovery/sitemap-lastmod-verifiabilityGrade AScoredSitemap lastmod values are verifiable against the pages
Cross-validates sampled sitemap <lastmod> values against three independent page-level modification signals — the Last-Modified response header, JSON-LD dateModified/datePublished, and article:modified_time — and scores agreement rather than presence. Detects the two dominant failure modes: the build stamp (every URL updated on every deploy) and the frozen value (the CMS never updates it).
machine-discovery/three-way-freshness-lagGrade BScoredThe sitemap and the feed are as fresh as the site itself
Compares the newest date the pages themselves publish against the newest `<lastmod>` in the sitemap and the newest entry in the feed. A surface that trails the site by more than a week is regenerated on a slower cadence than publication, so a crawler polling it sees a site that stopped publishing. Also checks that a feed’s own build timestamp is not older than its newest item, and that its items are in newest-first order.
machine-discovery/websub-hub-advertisementGrade CAdvisoryFeeds advertise a WebSub hub and exactly one canonical self link
Reads the WebSub discovery links on each feed — the `Link:` response headers first, as the specification requires, then the document — and checks the shape the W3C Recommendation asks for: exactly one absolute `rel=self` equal to the URL the feed was fetched from, and at least one `rel=hub` over HTTPS that answers a HEAD. Advisory only: no AI answer engine is documented as a WebSub subscriber, so this audit never affects the score.