{
  "$comment": "Source registry for Agent Lighthouse evidence dossiers. Every claim in docs/evidence cites an id from this file. accessed = the research pass that built the registry; verified = the date that URL was last resolved, taken from the dossier that cites it and falling back to the accessed date.",
  "accessed": "2026-08-20",
  "sources": [
    {
      "id": "S1",
      "title": "MCP Specification 2026-07-28 — Streamable HTTP Transport",
      "url": "https://modelcontextprotocol.io/specification/2026-07-28/basic/transports/streamable-http",
      "type": "spec",
      "publisher": "Model Context Protocol (Anthropic / MCP Working Groups)",
      "verified": "2026-08-20",
      "keyFindings": "Revision 2026-07-28 REMOVED the GET stream endpoint and protocol-level sessions (Mcp-Session-Id, Last-Event-ID). Server MUST expose one POST endpoint. Server MUST validate Origin; if Origin is present and invalid it MUST return 403 Forbidden. Every POST MUST carry MCP-Protocol-Version, Mcp-Method, and (for tools/call, resources/read, prompts/get) Mcp-Name headers; these are 'REQUIRED for compliance'. Header value MUST match the _meta body value or server MUST return 400 + JSON-RPC code -32020 HeaderMismatch. Unknown protocol version -> 400 + UnsupportedProtocolVersionError. Unknown method -> 404 + -32601. x-mcp-header constraints defined; clients MUST reject (exclude from tools/list) tools that violate them. Servers SHOULD send X-Accel-Buffering: no on SSE. GET/DELETE to endpoint SHOULD now return 405."
    },
    {
      "id": "S10",
      "title": "Lighthouse audit source: agent-accessibility-tree.js",
      "url": "https://raw.githubusercontent.com/GoogleChrome/lighthouse/main/core/audits/agentic/agent-accessibility-tree.js",
      "type": "repo",
      "publisher": "Google Chrome / Lighthouse",
      "verified": "2026-08-20",
      "keyFindings": "Implementation is a filter over artifacts.Accessibility.violations against ~37 TARGET_RULES from axe (button-name, link-name, input-button-name, label, autocomplete-valid, aria-allowed-attr, aria-required-attr, aria-valid-attr-value, tabindex, table/definition-list rules). Binary score: any violation scores 0. Crucially it inherits axe's blind spots — axe cannot fail an element that has no interactive semantics at all, and autocomplete-valid only validates tokens that are already present, never their absence."
    },
    {
      "id": "S11",
      "title": "WebSuite: Systematically Evaluating Why Web Agents Fail",
      "url": "https://arxiv.org/html/2406.01623v1",
      "type": "study",
      "publisher": "arXiv",
      "verified": "2026-08-20",
      "keyFindings": "Per-UI-primitive success rates for natbot and SeeAct. Worst patterns: slider interaction 0% for both agents; tooltip-based information retrieval 0% for both; complex form filling 12.5% (natbot) / 0% (SeeAct). Aggregate: operational actions 85.2%/76.2%, menu navigation 93.8%/81.3%, informational actions 43.8%/40.6%. Taxonomy covers click (button, link, icon button, slider, switch, accordion, dropdown menu, dialog button, snackbar), type (text/date/phone), select (checkbox, multicheck, select, datagrid row)."
    },
    {
      "id": "S12",
      "title": "Text fragments",
      "url": "https://web.dev/articles/text-fragments",
      "type": "vendor-doc",
      "publisher": "Google / web.dev",
      "verified": "2026-08-20",
      "keyFindings": "Confirms a shipped answer-surface consumer: \"Clicking a featured snippet takes the user directly to the featured snippet text on the source web page. This works thanks to automatically created Text Fragments URLs.\" Support: Chrome 89+, Edge 89+, Firefox 131+, Safari 18.2+. Restates the boundary rule: \"Each of prefix-, start, end, and -suffix can only match text within a single block-level element, but full start,end ranges can span multiple blocks.\" Opt-out header: Document-Policy: force-load-at-top."
    },
    {
      "id": "S13",
      "title": "MCP Specification (latest) — index",
      "url": "https://modelcontextprotocol.io/specification/latest",
      "type": "spec",
      "publisher": "Model Context Protocol",
      "verified": "2026-08-20",
      "keyFindings": "Confirms the current authoritative revision is 2026-07-28 (schema/2026-07-28/schema.ts). Lists optional extensions negotiated in capabilities: Tasks (io.modelcontextprotocol/tasks), MCP Apps (io.modelcontextprotocol/ui), Skills over MCP. Restates that annotations describing tool behavior 'should be considered untrusted, unless obtained from a trusted server'."
    },
    {
      "id": "S14",
      "title": "MCP Specification 2025-06-18 — Transports (superseded baseline)",
      "url": "https://modelcontextprotocol.io/specification/2025-06-18/basic/transports",
      "type": "spec",
      "publisher": "Model Context Protocol",
      "verified": "2026-08-20",
      "keyFindings": "The legacy-era shape used for backward-compat detection: Mcp-Session-Id assigned in the InitializeResult, GET opens a standalone SSE stream or returns 405, DELETE terminates a session, Last-Event-ID resumability, and servers SHOULD assume 2025-03-26 when MCP-Protocol-Version is absent. Also documents the deprecated 2024-11-05 HTTP+SSE detection path (GET returns an `endpoint` event as the first SSE event)."
    },
    {
      "id": "S15",
      "title": "browser-use DOM extraction: enhanced_snapshot.py",
      "url": "https://raw.githubusercontent.com/browser-use/browser-use/main/browser_use/dom/enhanced_snapshot.py",
      "type": "repo",
      "publisher": "Browser Use",
      "verified": "2026-08-20",
      "keyFindings": "Parses CDP DOMSnapshot for exactly these computed styles: display, visibility, opacity, overflow, overflow-x, overflow-y, cursor, pointer-events, position, background-color — plus bounding boxes, client rects, scroll rects, paint order and stacking contexts, and a CDP isClickable flag. Confirms production agents infer interactivity from cursor style and occlusion/paint order, so cursor:pointer-without-role and overlay occlusion are first-class, measurable inputs to a real agent's world model."
    },
    {
      "id": "S16",
      "title": "Playwright MCP server",
      "url": "https://github.com/microsoft/playwright-mcp",
      "type": "repo",
      "publisher": "Microsoft",
      "verified": "2026-08-21",
      "keyFindings": "Default mode is 'Playwright's accessibility tree, not pixel-based input'; browser_snapshot returns interactive elements with roles and accessible names, and every action tool takes a 'target' = 'exact target element reference from the page snapshot'. Coordinate clicking exists only behind the optional --caps=vision flag. Therefore an element absent from the a11y snapshot is literally unaddressable by the default toolchain."
    },
    {
      "id": "S17",
      "title": "Lighthouse overview — audit categories",
      "url": "https://developer.chrome.com/docs/lighthouse/overview",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-20",
      "keyFindings": "Confirms the shipped Agentic Browsing category composition: WebMCP integration (registered tools, forms missing declarative WebMCP support, schema validity), Discoverability (llms.txt), Accessibility for agents, Layout Stability. No DOM-operability, occlusion, control-substitution, or state-introspection auditing."
    },
    {
      "id": "S18",
      "title": "Why Do LLM-based Web Agents Fail? A Hierarchical Planning Perspective",
      "url": "https://arxiv.org/abs/2603.14248",
      "type": "study",
      "publisher": "arXiv",
      "verified": "2026-08-20",
      "keyFindings": "Decomposes failures across planning, execution and replanning layers and concludes 'low-level execution remains the dominant bottleneck', arguing that 'improving perceptual grounding and adaptive control, not only high-level reasoning, is critical'. Supports prioritising DOM-level operability checks over content/semantics checks when predicting agent task failure."
    },
    {
      "id": "S19",
      "title": "Operator System Card",
      "url": "https://cdn.openai.com/operator_system_card.pdf",
      "type": "vendor-doc",
      "publisher": "OpenAI",
      "verified": "2026-08-20",
      "keyFindings": "Documents confirmation prompts before 'actions that affect the state of the world (e.g., before completing a purchase or sending an email)' — 92% recall on 607 risky-action tasks, reducing mistake risk ~90% — plus watch mode forcing supervision on high-impact sites, and proactive refusal of high-risk categories. Implication for site authors: an agent must be able to observe and verify state transitions before and after acting, which requires persistent, machine-readable confirmation of what changed."
    },
    {
      "id": "S2",
      "title": "Playwright: Auto-waiting / Actionability checks",
      "url": "https://playwright.dev/docs/actionability",
      "type": "vendor-doc",
      "publisher": "Microsoft",
      "verified": "2026-08-20",
      "keyFindings": "Before click/check/fill/selectOption, Playwright enforces five checks: Visible (non-empty bounding box, not visibility:hidden), Stable (same bounding box over 2 animation frames), Receives Events (element is the hit target at the action point — overlays cause failure), Enabled (not [disabled]/aria-disabled), Editable (not readonly/aria-readonly). Fill requires visible+enabled+editable. This is the exact gate every Playwright-based agent (Playwright-MCP, browser-use, most CUA harnesses) passes through, so each check is a directly testable site-side failure cause."
    },
    {
      "id": "S3",
      "title": "MCP Specification 2026-07-28 — Versioning and Compatibility",
      "url": "https://modelcontextprotocol.io/specification/2026-07-28/basic/versioning",
      "type": "spec",
      "publisher": "Model Context Protocol",
      "verified": "2026-08-20",
      "keyFindings": "'There is no negotiation handshake.' Terminology: Modern = 2026-07-28+ (per-request _meta); Legacy = 2025-11-25 and earlier (initialize handshake). Unsupported version MUST return error code -32022 with data.supported[] and data.requested. Verbatim compatibility matrix: Modern client + Legacy server = FAILS. Legacy client + Modern server = FAILS. Only dual-era implementations bridge. Extensions negotiated via capabilities.extensions map with mandatory reverse-DNS prefix."
    },
    {
      "id": "S4",
      "title": "MCP Specification 2026-07-28 — Tools",
      "url": "https://modelcontextprotocol.io/specification/2026-07-28/server/tools",
      "type": "spec",
      "publisher": "Model Context Protocol",
      "verified": "2026-08-20",
      "keyFindings": "tools/list result set MUST NOT vary per-connection or as a side effect of other requests (MAY vary by authorization). Servers SHOULD return tools in deterministic order — rationale given verbatim: enables client caching and 'improves LLM prompt cache hit rates'. inputSchema MUST be a valid JSON Schema object (not null); defaults to JSON Schema 2020-12. Tool names SHOULD be 1-128 chars, case-sensitive, only [A-Za-z0-9_.-], unique within a server. Full x-mcp-header constraint list including static-reachability rule (chain of only `properties` keys; never through items/oneOf/anyOf/allOf/not/if/then/else/$ref). Clients MUST exclude violating tools from tools/list. If outputSchema present, servers MUST conform. Clients MUST treat annotations as untrusted."
    },
    {
      "id": "S5",
      "title": "MCP Specification 2026-07-28 — Caching",
      "url": "https://modelcontextprotocol.io/specification/2026-07-28/server/utilities/caching",
      "type": "spec",
      "publisher": "Model Context Protocol",
      "verified": "2026-08-20",
      "keyFindings": "'Servers MUST include caching hints on results with resultType: \"complete\"' for server/discover, tools/list, prompts/list, resources/list, resources/templates/list, resources/read. ttlMs is an integer ms; servers MUST provide ttlMs >= 0. If ttlMs is absent clients SHOULD assume 0 = immediately stale. cacheScope is exactly \"public\" or \"private\". Servers MUST apply the same cacheScope to all pages of a paginated list. Public scope on an authenticated endpoint may be shared across access tokens — servers MUST NOT rely on cacheScope for access control."
    },
    {
      "id": "S6",
      "title": "MCP Specification 2026-07-28 — Authorization",
      "url": "https://modelcontextprotocol.io/specification/2026-07-28/basic/authorization",
      "type": "spec",
      "publisher": "Model Context Protocol",
      "verified": "2026-08-20",
      "keyFindings": "'MCP servers MUST implement OAuth 2.0 Protected Resource Metadata (RFC9728).' Authorization servers MUST provide RFC8414 or OIDC Discovery. Servers SHOULD include a scope parameter in the WWW-Authenticate challenge. Example verbatim: `WWW-Authenticate: Bearer resource_metadata=\"https://mcp.example.com/.well-known/oauth-protected-resource\", scope=\"files:read\"`. Insufficient scope -> 403 with error=\"insufficient_scope\". Servers SHOULD NOT include offline_access in WWW-Authenticate scope or in PRM scopes_supported. Canonical server URI rules: no fragment, scheme required, prefer no trailing slash. Servers MUST validate token audience; MUST NOT accept or transit other tokens."
    },
    {
      "id": "S7",
      "title": "MCP Specification 2026-07-28 — Authorization Server Discovery",
      "url": "https://modelcontextprotocol.io/specification/2026-07-28/basic/authorization/authorization-server-discovery",
      "type": "spec",
      "publisher": "Model Context Protocol",
      "verified": "2026-08-20",
      "keyFindings": "PRM document returned by the MCP server MUST include authorization_servers with at least one entry (stronger than RFC 9728, where it is OPTIONAL). Two discovery mechanisms, both of which clients MUST support: WWW-Authenticate resource_metadata, then well-known probing in order — path-inserted (https://example.com/public/mcp -> https://example.com/.well-known/oauth-protected-resource/public/mcp) then root. AS metadata probing order for issuers with a path: /.well-known/oauth-authorization-server/{path}, /.well-known/openid-configuration/{path}, {path}/.well-known/openid-configuration; without a path: /.well-known/oauth-authorization-server then /.well-known/openid-configuration. Clients MUST reject a metadata doc whose issuer differs from the issuer used to build the URL."
    },
    {
      "id": "S8",
      "title": "MCP Security Best Practices (2026-07-28)",
      "url": "https://modelcontextprotocol.io/docs/2026-07-28/tutorials/security/security_best_practices.md",
      "type": "spec",
      "publisher": "Model Context Protocol",
      "verified": "2026-08-20",
      "keyFindings": "Token passthrough: 'MCP servers MUST NOT accept any tokens that were not explicitly issued for the MCP server.' Scope minimization: 'Common Mistakes' list names publishing all possible scopes in scopes_supported and using wildcard/omnibus scopes (*, all, full-access). State handle hijacking replaces session hijacking now that MCP is stateless: servers MUST NOT treat possession of a state handle as authentication; SHOULD use non-deterministic handles bound server-side to the authenticated user. SSRF section: clients SHOULD require HTTPS for all OAuth-related URLs and block private/link-local ranges (169.254.0.0/16 etc.)."
    },
    {
      "id": "S9",
      "title": "RFC 9728 — OAuth 2.0 Protected Resource Metadata",
      "url": "https://www.rfc-editor.org/rfc/rfc9728.html",
      "type": "spec",
      "publisher": "IETF",
      "verified": "2026-08-20",
      "keyFindings": "`resource` is the only REQUIRED metadata parameter; scopes_supported and resource_name are RECOMMENDED; authorization_servers is OPTIONAL at the RFC level. Section 3 well-known construction: insert /.well-known/oauth-protected-resource between host and path, removing any terminating slash after the host (https://resource.example.com/resource1 -> https://resource.example.com/.well-known/oauth-protected-resource/resource1). Section 3.3 validation: the retrieved `resource` value MUST be identical to the resource identifier used to build the request URL; on mismatch the response data MUST NOT be used. Section 7.7 recommends blocking private/reserved IP ranges."
    },
    {
      "id": "a2a-agent-discovery",
      "title": "A2A Protocol — Agent Discovery",
      "url": "https://a2a-protocol.org/latest/topics/agent-discovery/",
      "type": "spec",
      "publisher": "Linux Foundation / A2A Project",
      "verified": "2026-08-20",
      "keyFindings": "Normative path: `https://{agent-server-domain}/.well-known/agent-card.json`, following RFC 8615. Three discovery mechanisms defined: Well-Known URI, Curated Registries, Direct Configuration."
    },
    {
      "id": "a2a-repo",
      "title": "a2aproject/A2A",
      "url": "https://github.com/a2aproject/A2A",
      "type": "draft-spec",
      "publisher": "Linux Foundation / A2A Project",
      "verified": "2026-08-20",
      "keyFindings": "25,433 stars, 2,575 forks, created 2025-03-25, last push 2026-08-18 — the largest and most active agent-protocol repo in this domain by an order of magnitude over agents.json."
    },
    {
      "id": "a2a-spec",
      "title": "A2A Protocol Specification v1.0.0",
      "url": "https://a2a-protocol.org/latest/specification/",
      "type": "spec",
      "publisher": "Linux Foundation / A2A Project",
      "verified": "2026-08-20",
      "keyFindings": "Version 1.0.0. Agent Card is 'a JSON metadata document published by an A2A Server that describes its identity, capabilities, skills, service endpoint, and authentication requirements.' This URL is the IANA change-controller reference for the registered `agent-card.json` well-known suffix."
    },
    {
      "id": "acceptmarkdown-agent-matrix",
      "title": "Which AI agents support Markdown content negotiation? (status matrix)",
      "url": "https://acceptmarkdown.com/status",
      "type": "dataset",
      "publisher": "acceptmarkdown.com",
      "verified": "2026-08-20",
      "keyFindings": "Per-agent Accept-header matrix, tests Apr–Jun 2026, last updated 2026-06-22. Methodology: capture Accept headers in server logs, trigger agents at specific URLs, inspect access logs. SEND text/markdown (7): Claude Code ('Accept: text/markdown, text/html, */*'), Copilot Chat, Copilot CLI, Cursor, Microsoft Copilot, OpenClaw, OpenCode (q=1.0). PARTIAL — Codex CLI (OpenAI): no markdown Accept preference, but 'fetches the canonical URL as HTML first, then parses the response for <link rel=\"alternate\" type=\"text/markdown\" href=…>' and requests the markdown version separately. This is the ONLY named consumer of the rel=alternate link tag I found, and it is single-sourced. NO SUPPORT (HTML only): Aider, ChatGPT, Claude.ai, Cline, Devin, Gemini variants, Grok, Perplexity, v0, Windsurf, Zed."
    },
    {
      "id": "acp-architecture",
      "title": "ACP Concepts: Architecture",
      "url": "https://agenticcommerce.dev/docs/concepts/architecture",
      "type": "spec",
      "publisher": "Agentic Commerce Protocol",
      "verified": "2026-08-20",
      "keyFindings": "Four roles: Buyer, Agent, Seller (merchant of record, implements Checkout API), Payment Provider (tokenizes credentials with allowance constraints). IMPORTANT NEGATIVE RESULT: the architecture documents no seller discovery mechanism — no registry, no .well-known URL, no automatic endpoint discovery. Seller onboarding is out-of-band/manual. This means any 'ACP endpoint discovery' audit check would be speculative today, and endpoint conformance testing must accept an operator-supplied base URL."
    },
    {
      "id": "acp-capabilities",
      "title": "ACP Concepts: Capability Negotiation",
      "url": "https://agenticcommerce.dev/docs/concepts/capability-negotiation",
      "type": "spec",
      "publisher": "Agentic Commerce Protocol",
      "verified": "2026-08-20",
      "keyFindings": "Sellers advertise support via a `capabilities` object in API responses, not via HTTP headers. capabilities.payment.handlers identifies payment methods, PSP and instrument schemas. Interventions negotiated as the intersection of agent and seller support, with types including 3ds, biometric, address_verification. Extensions declared per-session. Same capabilities structure used by both parties; request-vs-response context indicates who is declaring."
    },
    {
      "id": "acp-checkout-openai",
      "title": "Agentic Checkout Specification",
      "url": "https://developers.openai.com/commerce/specs/checkout/",
      "type": "spec",
      "publisher": "OpenAI / Stripe (Agentic Commerce Protocol)",
      "verified": "2026-08-20",
      "keyFindings": "Five merchant-hosted HTTPS+JSON endpoints: POST /checkout_sessions (201), POST /checkout_sessions/{checkout_session_id}, POST /checkout_sessions/{id}/complete, POST /checkout_sessions/{id}/cancel (405 if not cancelable), GET /checkout_sessions/{id} (404 if absent). Request headers: Authorization, Accept-Language, User-Agent, Idempotency-Key, Request-Id, Content-Type, Signature, Timestamp (RFC 3339), API-Version. Response MUST echo Idempotency-Key and Request-Id. Session object carries id, status, currency (lowercase ISO 4217), line_items, fulfillment_options, totals, messages, links, payment_provider. Error envelope: {type, code, message, param(JSONPath)}. Message error codes include missing, invalid, out_of_stock, payment_declined, requires_sign_in, requires_3ds. Link types include terms_of_use, privacy_policy, seller_shop_policies."
    },
    {
      "id": "acp-checkout-spec",
      "title": "Agentic Commerce Protocol — Agentic Checkout Spec",
      "url": "https://developers.openai.com/commerce/specs/checkout",
      "publisher": "OpenAI",
      "type": "spec",
      "keyFindings": "Five REST endpoints: POST /checkout_sessions, POST /checkout_sessions/{id}, /complete, /cancel, GET /checkout_sessions/{id}. Scoped to checkout only, not search or catalog browsing. REST-only delivery; \"In the future, the Agentic Checkout Spec will support MCP servers\". No in-page tools, no HTML attributes, no WebMCP reference.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "acp-feed",
      "title": "Agentic Commerce Protocol — Product Feed Spec",
      "url": "https://agentic-commerce-protocol.com/docs/commerce/specs/feed",
      "type": "spec",
      "publisher": "OpenAI + Stripe",
      "verified": "2026-08-20",
      "keyFindings": "Three specs: Agentic Checkout, Product Feed, Delegated Payment (Apache 2.0). Feed required fields include seller_name, seller_url, seller_privacy_policy, seller_tos (last two required if checkout enabled), shipping as country:region:service_class:price. Recommended: offer_id, color, size, size_system, gender. Formats TSV/CSV/XML/JSON. Feeds are PUSHED to OpenAI at an agreed endpoint (not published at a public well-known URL), refresh every 15 min — so no external audit can fetch a merchant's ACP feed."
    },
    {
      "id": "acp-feed-openai",
      "title": "OpenAI Product Feed Specification",
      "url": "https://developers.openai.com/commerce/specs/feed/",
      "type": "vendor-doc",
      "publisher": "OpenAI",
      "verified": "2026-08-21",
      "keyFindings": "Required feed fields: item_id (<=100 chars), title (<=150), description (<=5000), brand (<=70), url, image_url, price (number + ISO 4217), availability enum {in_stock,out_of_stock,pre_order,backorder,unknown}, is_eligible_search, is_eligible_checkout, target_countries. CONDITIONAL hard requirements: availability_date required if availability is pre_order/backorder; seller_privacy_policy AND seller_tos required if is_eligible_checkout=true; gtin OR mpn required unless identifier_exists=no. Recommended: group_id, listing_has_variations, variant_dict, size, size_system, color, item_group_title, review_count, star_rating, q_and_a, reviews, age_restriction. Optional fulfillment field `shipping` uses format country:region:service_class:price:handling_days:transit_days. Returns fields: accepts_returns, return_deadline_in_days (positive int), accepts_exchanges, return_policy (URL, HTTPS preferred). sale_price must be less than price. Formats: .txt/.tsv/.csv (+.gz), UTF-8, lowercase underscore header row. Parser auto-selects between OpenAI schema and a Google-compatible profile."
    },
    {
      "id": "acp-feed-spec",
      "title": "Product Feed Specification — Agentic Commerce Protocol",
      "url": "https://agentic-commerce-protocol.com/docs/commerce/specs/feed",
      "type": "draft-spec",
      "publisher": "Agentic Commerce Protocol",
      "verified": "2026-08-20",
      "keyFindings": "Defines its own field taxonomy (item_group_title, color, size, offer_id, …) as \"a structured source of truth\"; does not reference schema.org vocabulary, GTIN, or Google Merchant Center compatibility. The emerging agentic-commerce standard is deliberately feed-native, not markup-native."
    },
    {
      "id": "acp-github",
      "title": "Agentic Commerce Protocol (ACP) specification repository",
      "url": "https://github.com/agentic-commerce-protocol/agentic-commerce-protocol",
      "type": "draft-spec",
      "publisher": "OpenAI + Stripe (Founding Maintainers)",
      "verified": "2026-08-20",
      "keyFindings": "Apache 2.0, date-versioned releases (latest stable 2026-04-17): openapi.agentic_checkout.yaml, openapi.delegate_payment.yaml, JSON schemas; cart, feed, orders, auth and MCP surfaces. CRITICAL FOR SIGNAL 9: the repo defines NO website-side discovery mechanism — no well-known URL, no HTML link tag, no sitemap-based or markup-based endpoint discovery. Merchant endpoints are registered out-of-band."
    },
    {
      "id": "acp-home",
      "title": "Agentic Commerce Protocol homepage",
      "url": "https://agenticcommerce.dev/",
      "type": "spec",
      "publisher": "OpenAI + Stripe",
      "verified": "2026-08-20",
      "keyFindings": "Open standard (Apache 2.0) for programmatic commerce flows between buyers, AI agents and businesses, developed jointly by Stripe and OpenAI. Confirms the protocol is genuinely open and implementable by any merchant, not gated to a single platform."
    },
    {
      "id": "acp-openapi-cart",
      "title": "openapi.cart.yaml (spec version 2026-04-17)",
      "url": "https://raw.githubusercontent.com/agentic-commerce-protocol/agentic-commerce-protocol/main/spec/2026-04-17/openapi/openapi.cart.yaml",
      "type": "spec",
      "publisher": "Agentic Commerce Protocol",
      "verified": "2026-08-20",
      "keyFindings": "Seller-hosted pre-checkout Cart API at https://seller.example.com: POST /carts, GET /carts/{id}, PUT /carts/{id} (full replacement), POST /carts/{id}/cancel. Required headers Authorization (Bearer) and API-Version (e.g. 2026-04-17); Idempotency-Key required on POST; Content-Type application/json on POST/PUT; optional Accept-Language, Request-Id. The cart response carries a `continue_url` (e.g. https://seller.example.com/cart/cart_abc123) used to hand the buyer off to the seller's own checkout UI. No payment configuration in this phase; expired carts return 404. This is the lower-lift adoption path versus full delegated-payment Instant Checkout."
    },
    {
      "id": "acp-openapi-checkout",
      "title": "openapi.agentic_checkout.yaml (spec version 2026-04-17)",
      "url": "https://raw.githubusercontent.com/agentic-commerce-protocol/agentic-commerce-protocol/main/spec/2026-04-17/openapi/openapi.agentic_checkout.yaml",
      "type": "spec",
      "publisher": "Agentic Commerce Protocol",
      "verified": "2026-08-20",
      "keyFindings": "CheckoutSession REQUIRED fields (9): id, status, currency, line_items, totals, fulfillment_options, messages, links, capabilities. links[].type enum expanded to 8 values: terms_of_use, privacy_policy, return_policy, shipping_policy, contact_us, about_us, faq, support. status enum (11): incomplete, not_ready_for_payment, requires_escalation, authentication_required, ready_for_payment, pending_approval, complete_in_progress, completed, canceled, in_progress, expired. totals[].type enum (12): items_base_amount, items_discount, subtotal, discount, fulfillment, tax, fee, gift_wrap, tip, store_credit, total, amount_refunded. Message error codes extended with low_stock, quantity_exceeded, coupon_invalid, coupon_expired, minimum_not_met, maximum_exceeded, region_restricted, age_verification_required, approval_required, unsupported, not_found, conflict, rate_limited, expired, intervention_required. API-Version is YYYY-MM-DD, required on all requests. Response headers Idempotency-Key and Request-Id are required echoes."
    },
    {
      "id": "acp-openapi-feed",
      "title": "openapi.feed.yaml (spec version 2026-04-17)",
      "url": "https://raw.githubusercontent.com/agentic-commerce-protocol/agentic-commerce-protocol/main/spec/2026-04-17/openapi/openapi.feed.yaml",
      "type": "spec",
      "publisher": "Agentic Commerce Protocol",
      "verified": "2026-08-20",
      "keyFindings": "Merchant-hosted feed API (server https://merchant.example.com): POST /feeds, GET /feeds/{id}, GET /feeds/{id}/products, PATCH /feeds/{id}/products. Product object REQUIRES id and variants[] (array of Variant). Variant REQUIRES id and title; optional description, url, barcodes, price, list_price, unit_price, availability, categories, condition, variant_options, media, seller, marketplace. Critically, the model is variant-first: every sellable thing is a Variant with its own id, price and availability — a product without resolvable variants has no purchasable unit."
    },
    {
      "id": "acp-payment",
      "title": "Delegated Payment Specification",
      "url": "https://developers.openai.com/commerce/specs/payment.md",
      "type": "spec",
      "publisher": "OpenAI / Stripe",
      "verified": "2026-08-20",
      "keyFindings": "PSP-or-merchant-hosted endpoint POST /agentic_commerce/delegate_payment. Same nine headers as checkout (Authorization, Accept-Language, User-Agent, Idempotency-Key, Request-Id, Content-Type, Signature, Timestamp, API-Version). Request requires payment_method (type card only; card number type fpan or network_token) and allowance {reason:'one_time', max_amount, currency (ISO 4217), checkout_session_id, merchant_id, expires_at (RFC 3339)}; optional billing_address, risk_signals, metadata. Success 201 returns id = vault token in the form vt_… — single-use, scoped outside PCI scope, bounded by max_amount and expiry."
    },
    {
      "id": "acp-repo",
      "title": "agentic-commerce-protocol repository",
      "url": "https://github.com/agentic-commerce-protocol/agentic-commerce-protocol",
      "type": "repo",
      "publisher": "Agentic Commerce Protocol (OpenAI + Stripe, Apache 2.0)",
      "verified": "2026-08-20",
      "keyFindings": "Date-versioned spec releases: 2025-09-29 (initial), 2025-12-12 (fulfillment), 2026-01-16 (capability negotiation), 2026-01-30 (extensions, discounts, payment handlers), 2026-04-17 (current stable: cart, feed, orders, authentication, MCP), plus unreleased/. Artifacts under spec/<version>/: openapi/, json-schema/, openrpc/. openapi/ for 2026-04-17 contains openapi.agentic_checkout.yaml, openapi.agentic_checkout_webhook.yaml, openapi.cart.yaml, openapi.delegate_authentication.yaml, openapi.delegate_payment.yaml, openapi.feed.yaml. No .well-known or discovery mechanism anywhere in the repo."
    },
    {
      "id": "acp-security",
      "title": "ACP Concepts: Security",
      "url": "https://agenticcommerce.dev/docs/concepts/security",
      "type": "spec",
      "publisher": "Agentic Commerce Protocol",
      "verified": "2026-08-20",
      "keyFindings": "All ACP endpoints use HTTPS and send/receive JSON. Bearer-token authentication between agent and seller; sellers retrieve tokens through the agent's application. Mandatory headers: Authorization: Bearer <token>, Content-Type: application/json, Accept: application/json."
    },
    {
      "id": "acp-sellers",
      "title": "ACP Getting Started: Sellers",
      "url": "https://agenticcommerce.dev/docs/getting-started/sellers",
      "type": "spec",
      "publisher": "Agentic Commerce Protocol",
      "verified": "2026-08-20",
      "keyFindings": "Seller obligations: implement the five HTTPS/JSON checkout endpoints; calculate all amounts (item prices, discounts, taxes, shipping); manage inventory and availability; process payments through their PSP; fulfil orders. Sellers must declare capabilities in EVERY checkout response, including payment handlers (handler id, name, version, PSP reference, configuration) and optional extensions. Sellers must validate payment handler IDs against declared capabilities and manage state transitions to ready_for_payment."
    },
    {
      "id": "acp-spec-confirm",
      "title": "Agentic Commerce Protocol",
      "url": "https://www.agenticcommerce.dev/",
      "publisher": "OpenAI + Stripe",
      "type": "spec",
      "keyFindings": "Completion/confirmation flows run through agent-facing endpoints ('Publish your checkout configuration with a traditional API or MCP') integrated with the merchant's commerce backend and payment processor. No schema.org, no confirmation-page markup.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "aeo-auditor-long-tail",
      "title": "aeo-auditor ('Think Lighthouse, but for AI coding agents')",
      "url": "https://github.com/JhontyBarreto/aeo-auditor",
      "type": "article",
      "publisher": "JhontyBarreto — 0 stars",
      "verified": "2026-08-20",
      "keyFindings": "Representative of a very large 0-star long tail confirmed via the GitHub search API in this session: AJ-EN/aeo-audit ('PageSpeed for AI Agents'), saai07/GeoLens, devakrishnar/AEO-Analyzer, akimovpro/seo-for-ai, api-evangelist/gleefulai-visibility, truos-official/Generative-Engine-Optimization-Audit, Third-Sun-Pro/geo-auditor, msiroya21/GEO-Auditor, echorank-com/geo-checker-cli, indexcraft/ai-crawler-access-toolkit, plus a dozen llms.txt generator/validator kits. Near-identical check surfaces (llms.txt, robots.txt AI-bot rules, structured data, content structure), near-zero adoption, and not one publishes evidence grading. The category is crowded but undifferentiated on rigour."
    },
    {
      "id": "agent-lighthouse-own-measurement",
      "title": "Direct measurement of vendor llms.txt files and markdown content negotiation (2026-08-20)",
      "url": "https://llmstxt.org/",
      "type": "dataset",
      "publisher": "Agent Lighthouse (this research, reproducible via curl)",
      "verified": "2026-08-21",
      "keyFindings": "ORIGINAL EMPIRICAL DATA, reproducible. (1) CONTENT NEGOTIATION LIVE: curl -H 'Accept: text/markdown' returned 'content-type: text/markdown; charset=utf-8' from platform.claude.com/docs, docs.stripe.com/api, and developers.cloudflare.com/fundamentals/. (2) LINK TAG LIVE: developers.cloudflare.com emits <link rel=\"alternate\" type=\"text/markdown\" href=\"https://developers.cloudflare.com/fundamentals/index.md\">; docs.stripe.com and nextjs.org/docs emit none. Stripe instead emits Link: </.well-known/skills/index.json>; rel=\"service-meta\". (3) llms-full.txt SIZES ARE ABSURD: Anthropic 33.5 MB, Cloudflare 57.3 MB, OpenAI 6.1 MB, Perplexity 4.1 MB; Stripe returns 404. These exceed every production context window by 1-2 orders of magnitude. (4) FORMAT CONFORMANCE across 7 major vendor llms.txt files (Anthropic, Cloudflare, Perplexity, Stripe, OpenAI, GitHub, Vercel): 7/7 have an H1 and markdown links (all would PASS Lighthouse), but only 5/7 have a blockquote summary — Anthropic and Stripe omit it. Link counts range 38 (OpenAI) to 1,872 (Vercel); sizes 5.8 KB to 215 KB."
    },
    {
      "id": "agent-ready-websites",
      "title": "Designing Agent-Ready Websites for AI Web Agents",
      "url": "https://arxiv.org/abs/2607.12056",
      "publisher": "arXiv",
      "type": "study",
      "verified": "2026-08-20",
      "keyFindings": "Structures agent readiness as 'agent interpretability, agent executability, and agent decision reliability', supported by 'machine readability, semantic clarity, agent actionability, and contextual decision-reliability signals', with improvements in 'structural clarity, action cues, evidence signals, and temporal validity indicators'. Framework identity, framework detection and client-side rendering are absent from the framework; it also notes GEO metrics do not fully assess agent-readiness.",
      "accessed": "2026-08-21"
    },
    {
      "id": "agentic-commerce-protocol",
      "title": "Agentic Commerce Protocol",
      "url": "https://www.agenticcommerce.dev/",
      "publisher": "OpenAI + Stripe",
      "type": "spec",
      "keyFindings": "'An open standard for programmatic commerce flows between buyers, AI agents, and businesses.' Merchants 'Publish your checkout configuration with a traditional API or MCP'. No schema.org markup, no JSON-LD, no potentialAction anywhere in the protocol.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "agents-json-com-dead",
      "title": "agents-json.com — upstream repo's listed homepage, unreachable",
      "url": "https://agents-json.com",
      "publisher": "Wild Card AI",
      "type": "vendor-doc",
      "keyFindings": "(Domain does not resolve — the dead domain IS the evidence.) DNS resolves to 216.92.3.51 but both HTTPS and HTTP time out (curl exit 28, http_code=000); WebFetch returned ECONNREFUSED. The project has no live website at either of its two domains.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "agents-txt-spec",
      "title": "agents.txt specification",
      "url": "https://agents-txt.com/spec/",
      "publisher": "agents-txt.com (community)",
      "type": "spec",
      "keyFindings": "Only found source placing security.txt in an agent context: \"security.txt is a human-readable vulnerability disclosure channel published at /.well-known/security.txt; agents.txt is a machine-readable capability declaration. Sites that take agent payments or authentication SHOULD publish both.\" Does not reference Referrer-Policy, Permissions-Policy or preconnect. Community-maintained, no vendor adoption.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "agentsjson-org-dead",
      "title": "agentsjson.org — audit's docsUrl, domain no longer resolves",
      "url": "https://agentsjson.org/",
      "publisher": "Wild Card AI (defunct)",
      "type": "vendor-doc",
      "keyFindings": "(Domain does not resolve — the dead domain IS the evidence.) WebFetch failed with getaddrinfo ENOTFOUND; dig returns no A record; curl returns 'Could not resolve host: agentsjson.org' (http_code=000). This is the exact URL the audit ships as its docsUrl, so users are sent to a dead domain.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "agentswelcome",
      "title": "AGENTS WELCOME",
      "url": "https://agentswelcome.dev/",
      "publisher": "agentswelcome.dev (demo site)",
      "type": "article",
      "keyFindings": "Agent-first demo site listing security.txt as part of a \"complete machine-discovery surface\" alongside sitemap.xml and Atom feeds, while assigning the authoritative machine-readable role to /.well-known/agents.json. Self-describes as a demonstration, not a standard. No claim that agents parse security.txt.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "ahrefs",
      "title": "Ahrefs Brand Radar",
      "url": "https://ahrefs.com/brand-radar",
      "type": "vendor-doc",
      "publisher": "Ahrefs",
      "verified": "2026-08-20",
      "keyFindings": "AI visibility tracking, competitive benchmarking, AI citation discovery, YouTube/TikTok/Reddit monitoring, custom prompts, dashboard. Explicitly no llms.txt, no AI-bot robots.txt rules, no agent schema audits."
    },
    {
      "id": "ahrefs-ai-traffic-study",
      "title": "AI traffic study: how much traffic do AI chatbots actually send?",
      "url": "https://ahrefs.com/blog/ai-traffic-study/",
      "type": "study",
      "publisher": "Ahrefs",
      "verified": "2026-08-20",
      "keyFindings": "3,000 websites analyzed across seven AI platforms (ChatGPT, Claude, Copilot, Gemini, Perplexity, Jasper, Mistral), segmented by monthly visitor volume. 63% of sampled sites receive at least one AI-sourced visit. But AI chatbots generate only 0.12% of average website views and 0.17% of visitors. ChatGPT, Perplexity and Gemini are 98% of AI referrals; ChatGPT alone 50%. Small sites capture proportionally more (0.56% of visitors) than large ones (0.13%). Caveat: Copilot stopped passing referrer data around October 2024, so some AI traffic is misattributed to direct — real figures are somewhat higher."
    },
    {
      "id": "ahrefs-aio-top10",
      "title": "Update: 38% of AI Overview Citations Pull From The Top 10",
      "url": "https://ahrefs.com/blog/ai-overview-citations-top-10/",
      "type": "study",
      "publisher": "Ahrefs",
      "verified": "2026-08-20",
      "keyFindings": "863K keyword SERPs and 4M AI Overview URLs, data current as of 2 March 2026 (>2x the July 2025 study). Only 38% of AI-Overview-cited pages rank in the top 10 for the query; 31.2% rank 11–100 and 31.0% rank beyond 100. Organic blue links only: 37.1% / 26.2% / 36.7%. Down sharply from ~76% top-10 in July 2025. Attributed to query fan-out — sources recurring across simulated sub-queries get cited rather than those ranking for the direct query. YouTube is the single most-cited domain, supplying 18.2% of citations from outside the top 100."
    },
    {
      "id": "ahrefs-brand-correlation",
      "title": "An Analysis of AI Overview Brand Visibility Factors (75K Brands Studied)",
      "url": "https://ahrefs.com/blog/ai-overview-brand-correlation/",
      "type": "study",
      "publisher": "Ahrefs",
      "verified": "2026-08-20",
      "keyFindings": "75,000 brands (Domain Rating >40, highest-volume keyword ≥800 monthly searches); millions of AI Overview responses via Brand Radar; ~26% of domains had zero mentions. Spearman correlations with AI Overview brand visibility: brand web mentions 0.664, brand anchors 0.527, brand search volume 0.392, Domain Rating 0.326, referring domains 0.295, branded organic traffic 0.274, backlinks 0.218, URL Rating 0.18, site pages 0.17. Off-site brand signals beat backlinks by ~3x. Explicit caveats: 'correlation ≠ causation' and 'all the factors we studied revealed moderate to very weak correlations'. No author-level or on-page brand-consistency variable was measured."
    },
    {
      "id": "ahrefs-brand-radar",
      "title": "Ahrefs Brand Radar",
      "url": "https://ahrefs.com/brand-radar",
      "type": "vendor-doc",
      "publisher": "Ahrefs",
      "verified": "2026-08-20",
      "keyFindings": "Tracks brand mentions across AI Overviews/AI Mode, ChatGPT, Copilot, Gemini, Perplexity and Grok against a database of '476M+ total monthly prompts', described as 'search-backed prompts, not synthetic ones', and indexes YouTube/TikTok/Reddit as citation sources. Explicitly a visibility monitor, not a technical AI-readiness auditor — site-side checking is left to the separate Site Audit product, which has no AI-crawler audit surface described here."
    },
    {
      "id": "ahrefs-brand-visibility-correlations",
      "title": "Top Brand Visibility Factors in ChatGPT, AI Mode, and AI Overviews (75k Brands Studied)",
      "url": "https://ahrefs.com/blog/ai-brand-visibility-correlations/",
      "type": "study",
      "publisher": "Ahrefs",
      "verified": "2026-08-20",
      "keyFindings": "75,000 brands, millions of AI responses, Spearman correlations across ChatGPT / AI Mode / AI Overviews. Structured data and schema markup are NOT among the measured factors. Top correlates: YouTube mentions (~0.737), branded web mentions (0.656–0.709), branded anchors (0.511–0.628), branded search volume (0.352–0.466). Relevant as an absence-of-evidence datapoint: the largest AI-visibility correlation study did not consider schema worth measuring."
    },
    {
      "id": "ahrefs-freshness-17m",
      "title": "Do AI Assistants Prefer to Cite Fresh Content?",
      "url": "https://ahrefs.com/blog/do-ai-assistants-prefer-to-cite-fresh-content",
      "type": "study",
      "publisher": "Ahrefs",
      "verified": "2026-08-21",
      "keyFindings": "16.975M cited URLs across ChatGPT, Perplexity, Gemini, Copilot, Google AI Overviews and organic Google SERPs (Brand Radar + Content Explorer). AI-cited pages average 1,064 days old (2.9 years) vs 1,432 days for organic results = 25.7% fresher. Per-platform average age since publication: ChatGPT 958 days (strongest recency bias), Copilot 1,056, Gemini 1,118, Perplexity 1,166, Google AI Overviews 1,432 — AI Overviews are NOT fresher than organic. Author caveats: the average cited page is still 2.9 years old, so AI assistants still favour established content; warns against updating without meaningful change, noting 'low-quality, irrelevant content that's updated every day will not have a magic positive effect.'"
    },
    {
      "id": "ahrefs-schema-ai-citations",
      "title": "We Tracked 1,885 Pages Adding Schema. AI Citations Barely Moved.",
      "url": "https://ahrefs.com/blog/schema-ai-citations/",
      "type": "study",
      "publisher": "Ahrefs",
      "verified": "2026-08-21",
      "keyFindings": "Quasi-experiment: 1,885 pages that added JSON-LD Aug 2025–Mar 2026, matched 1:3 against ~4,000 never-JSON-LD control URLs on different domains with similar pre-period citation levels; 30-day pre/post windows; four tests (two-sample t-test, matched difference-in-differences, event study, alternate-window robustness). Results: Google AI Mode +2.4% (indistinguishable from zero), ChatGPT +2.2% (indistinguishable from zero), Google AI Overviews −4.6% (significant, ~1-in-2,500 chance of randomness, ≈12 daily citations per page). Schema types pooled (Article, FAQ, Product, HowTo, Organization) — no per-type isolation. Authors' caveats: sample restricted to pages already heavily cited (100+ AIO citations in Feb 2025), so it cannot speak to first-time discovery/parsing/indexing effects; cannot separate schema from concurrent page changes."
    },
    {
      "id": "ahrefs-short-vs-long-aio",
      "title": "Short vs. Long Content in AI Overviews: The Data Says Both Work",
      "url": "https://ahrefs.com/blog/short-vs-long-content-in-ai-overviews/",
      "type": "dataset",
      "publisher": "Ahrefs",
      "verified": "2026-08-20",
      "keyFindings": "560,346 AI Overviews, 1,677,876 cited URLs, 174,048 pages with valid data after boilerplate removal. 53.4% of citations go to pages under 1,000 words; 36.8% are 350–1,000 words; only 16% exceed 2,000 words. 'Near-zero correlation between word length and being cited' (Spearman 0.04); average word counts for citation positions 1–3 varied only 1,270–1,291. Median word count by type: product listings 315, blog articles 1,166, podcast audio 1,226. Conclusion favors structure over length: 'answer the query directly', use clear headings, 'write for humans first'. Establishes that raw length is not a citation lever."
    },
    {
      "id": "ai-catalog-consuming",
      "title": "Consuming catalogs guide",
      "url": "https://raw.githubusercontent.com/Agent-Card/ai-catalog/main/docs/guides/consuming-catalogs.md",
      "publisher": "Linux Foundation / Agent Card Working Group",
      "type": "spec",
      "keyFindings": "Prescribes consumer discovery order: HTTP Link header rel=\"ai-catalog\", then HTML <link rel=\"ai-catalog\" href=...>, then fall back to GET /.well-known/ai-catalog.json. Names no deployed consumers.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "ai-catalog-io",
      "title": "AI Catalog — Common AI Catalog and Registry Standard",
      "url": "https://ai-catalog.io/",
      "publisher": "Linux Foundation / Agent Card Working Group",
      "type": "spec",
      "keyFindings": "Defines the AI Catalog JSON container published at /.well-known/ai-catalog.json; maintained by the Agent Card WG under the Linux Foundation. Names no concrete consumers or adopters.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "ai-catalog-io-meta",
      "title": "ai-catalog.io specification site",
      "url": "https://ai-catalog.io/",
      "publisher": "Linux Foundation / Agent Card Working Group",
      "type": "spec",
      "keyFindings": "Describes the catalog as a JSON container for MCP servers, A2A agents, Claude Code plugins, datasets and model cards with optional Trust Manifest extensions for attestation/provenance — the spec's answer to 'who owns this', in place of owner/contact fields.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "ai-catalog-readme",
      "title": "Agent-Card/ai-catalog README",
      "url": "https://raw.githubusercontent.com/Agent-Card/ai-catalog/main/README.md",
      "publisher": "Linux Foundation (Agent Card WG)",
      "type": "repo",
      "keyFindings": "Calls itself a temporary Linux Foundation working repo; states A2A and MCP steering committees will vote on adoption — i.e. pre-ratification. 210 stars, actively pushed 2026-08-20.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "ai-catalog-repo",
      "title": "Agent-Card/ai-catalog — working repository for common AI Card standard",
      "url": "https://github.com/Agent-Card/ai-catalog",
      "type": "draft-spec",
      "publisher": "Agent-Card working group",
      "verified": "2026-08-20",
      "keyFindings": "210 stars, created 2025-10-29, last push 2026-08-20 (active). This is the protocol-agnostic catalog format that SEP-2127 delegates domain-level discovery to. Not an IETF/W3C document; no IANA registration for `ai-catalog.json`."
    },
    {
      "id": "allenai-crawler",
      "title": "AI2Bot — Allen Institute for AI crawler",
      "url": "https://allenai.org/crawler",
      "type": "vendor-doc",
      "publisher": "Allen Institute for AI",
      "verified": "2026-08-20",
      "keyFindings": "UA: 'Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler)'. Crawls 'to find web content' that is 'used to train open language models'. Page offers the UA string so operators 'can be used to filter or reject traffic from our crawler if desired'. PARTIAL NEGATIVE: the fetched page does not contain an explicit robots.txt-compliance sentence or a canonical disallow snippet."
    },
    {
      "id": "almanac-markup-2024",
      "title": "Web Almanac 2024 — Markup",
      "url": "https://almanac.httparchive.org/en/2024/markup",
      "type": "dataset",
      "publisher": "HTTP Archive",
      "verified": "2026-08-20",
      "keyFindings": "Median 594 elements per mobile page (p90 1,716); median HTML transfer size 33 kB desktop / 32 kB mobile; 10.5% of mobile pages serve HTML uncompressed; 86% of mobile pages contain at least one HTML comment and 26% still ship IE conditional comments; SVG present on 51.6% of pages. Population baseline for calibrating per-page token budgets and for the claim that dead markup ships at scale."
    },
    {
      "id": "almanac-page-weight-2024",
      "title": "Web Almanac 2024 — Page Weight",
      "url": "https://almanac.httparchive.org/en/2024/page-weight",
      "type": "dataset",
      "publisher": "HTTP Archive",
      "verified": "2026-08-20",
      "keyFindings": "Median total page 2,652 kB desktop / 2,311 kB mobile; p90 8,375 kB / 7,680 kB; median desktop homepage loads ~18 kB of HTML against ~1,054 kB images and ~613 kB JS. Useful contrast: for a rendering browser HTML is ~1% of weight, but for a non-rendering AI crawler that HTML document is ~100% of what gets tokenized — so HTML-internal waste is the entire agent-side cost."
    },
    {
      "id": "amazon-science-rufus",
      "title": "The technology behind Amazon's GenAI-powered shopping assistant, Rufus",
      "url": "https://www.amazon.science/blog/the-technology-behind-amazons-genai-powered-shopping-assistant-rufus",
      "type": "article",
      "publisher": "Amazon Science",
      "verified": "2026-08-20",
      "keyFindings": "Rufus data sources: the entire Amazon catalogue, customer reviews, community Q&A posts, \"public information on the web\", plus queries to internal systems and Stores APIs. No mention of schema.org, structured data formats, or crawling merchant markup. Counter-evidence: optimizing site Product markup has no documented path into Rufus."
    },
    {
      "id": "amazonbot-docs",
      "title": "Amazonbot",
      "url": "https://developer.amazon.com/amazonbot",
      "type": "vendor-doc",
      "publisher": "Amazon",
      "verified": "2026-08-20",
      "keyFindings": "UA: 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36'. 'Amazonbot is used to improve our products and services' and 'may be used to train Amazon AI models'. 'Automated crawling from these listed user agents respects the Robots Exclusion Protocol, honoring the user-agent and the allow/disallow directives.' Explicit negative on crawl-delay: 'They do not support the crawl-delay directive.' Also documents a separate Amzn-SearchBot which 'does not crawl content for generative AI model training'."
    },
    {
      "id": "anthropic-bots",
      "title": "Anthropic — Does Anthropic crawl data from the web?",
      "url": "https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler",
      "type": "vendor-doc",
      "publisher": "Anthropic",
      "verified": "2026-08-21",
      "keyFindings": "Three tokens with distinct purposes: ClaudeBot (training), Claude-User (live user-initiated fetch), Claude-SearchBot (search quality). IP list at https://claude.com/crawling/bots.json. Anthropic states IP-based blocking 'may not work correctly or persistently guarantee an opt-out' — robots.txt product tokens are the sanctioned control surface."
    },
    {
      "id": "anthropic-bots-json",
      "title": "Anthropic crawler IP prefixes (bots.json)",
      "url": "https://claude.com/crawling/bots.json",
      "type": "dataset",
      "publisher": "Anthropic",
      "verified": "2026-08-20",
      "keyFindings": "Live JSON with creationTime (observed 2026-08-18T23:56:36Z) and a 'prefixes' array of IPv4 CIDRs (e.g. 216.73.216.0/22, plus many /32s). Confirms Anthropic now publishes verifiable crawler IP ranges, enabling WAF allow-listing without UA spoofing risk."
    },
    {
      "id": "anthropic-browser-use-address",
      "title": "Browser use tool",
      "url": "https://platform.claude.com/docs/en/docs/agents-and-tools/tool-use/browser-use-tool",
      "publisher": "Anthropic",
      "type": "vendor-doc",
      "keyFindings": "Documents that the agent reads the page as an accessibility tree via `read_page`. Combined with the snapshot experiment, this establishes that the representation Claude receives contains no <address>-specific signal to act on.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "anthropic-browser-use-aside",
      "title": "Browser use tool",
      "url": "https://platform.claude.com/docs/en/docs/agents-and-tools/tool-use/browser-use-tool",
      "publisher": "Anthropic",
      "type": "vendor-doc",
      "keyFindings": "`read_page` returns 'the page's accessibility tree as text with each element tagged with a reference'; the doc instructs developers to prefer accessibility-tree references over pixel coordinates. Establishes the named vendor agent that receives the `complementary` landmark my experiment observed.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "anthropic-browser-use-images",
      "title": "Browser use tool",
      "url": "https://platform.claude.com/docs/en/docs/agents-and-tools/tool-use/browser-use-tool",
      "publisher": "Anthropic",
      "type": "vendor-doc",
      "keyFindings": "Confirms `read_page` returns the page's accessibility tree as text — the representation in which empty-alt images are already absent, so the audit's 'agents flag false content gaps' scenario has no node to flag.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "anthropic-browser-use-tool",
      "title": "Browser use tool (browser_toolset_20260801)",
      "url": "https://platform.claude.com/docs/en/agents-and-tools/tool-use/browser-use-tool",
      "type": "vendor-doc",
      "publisher": "Anthropic",
      "verified": "2026-08-21",
      "keyFindings": "First-party proof that Claude's browser agent perceives pages via the accessibility tree, not raw DOM or pixels. read_page: 'Return the page's accessibility tree as text with each element tagged with a reference such as [ref_2]'; filter=interactive returns only visible interactive elements; depth default 15; output capped at 50,000 characters and truncated. find: 'Search for elements matching a natural-language description such as \"search field\" or \"add to cart button\"'. form_input sets a form element's value by ref. get_page_text: 'Return the page's visible text as plain text, prioritizing the main article content'. Security guidance: 'build page reads from what the page renders (the accessibility tree or visible text), not raw DOM source, so hidden text doesn't reach Claude.' Element references come from read_page and find. Same tool surface is exposed by Anthropic's Claude-in-Chrome MCP server (read_page = 'Get an accessibility tree representation of elements on the page')."
    },
    {
      "id": "anthropic-chrome",
      "title": "Piloting Claude for Chrome",
      "url": "https://claude.com/blog/claude-for-chrome",
      "type": "vendor-doc",
      "publisher": "Anthropic",
      "verified": "2026-08-20",
      "keyFindings": "Red-team attack success rate 23.6% in autonomous browsing mode, 11.2% after mitigations; a browser-specific challenge set went 35.7% -> 0%. Names the exact vectors: 'hidden malicious form fields in a webpage's Document Object Model (DOM) invisible to humans, and other hard-to-catch injections such as through the URL text and tab title that only an agent might see.' This is the vendor-documented basis for auditing hidden inputs and a11y/metadata attributes."
    },
    {
      "id": "anthropic-citations-docs",
      "title": "Citations — Claude Platform Documentation",
      "url": "https://platform.claude.com/docs/en/docs/build-with-claude/citations",
      "type": "vendor-doc",
      "publisher": "Anthropic",
      "verified": "2026-08-20",
      "keyFindings": "Documents the citation unit for a major model provider: 'Document contents are \"chunked\" to define the minimum granularity of possible citations. For example, sentence chunking lets Claude cite a single sentence or chain together multiple consecutive sentences to cite a paragraph or longer passage.' 'For plain text documents: Content is chunked into sentences that can be cited from.' 'For PDFs: ... content is chunked into sentences.' 'For custom content documents: Your provided content blocks are used as-is and no further chunking is done.' Citation formats: character indices (plain text), page numbers (PDF), block indices (custom). Confirms the extraction unit is the sentence, not the section or the page."
    },
    {
      "id": "anthropic-claude-code-tools-reference",
      "title": "Tools reference — Claude Code docs (WebFetch behavior)",
      "url": "https://code.claude.com/docs/en/tools-reference",
      "type": "vendor-doc",
      "publisher": "Anthropic",
      "verified": "2026-08-20",
      "keyFindings": "STRONGEST A-GRADE VENDOR STATEMENT IN THIS DOMAIN, and it is about markdown, not llms.txt: \"WebFetch sets a User-Agent header beginning with Claude-User, and an Accept header that prefers Markdown over HTML so servers that support content negotiation can return Markdown directly.\" This is an explicit vendor commitment that a named agent consumes markdown alternates via content negotiation. No equivalent statement exists anywhere for llms.txt."
    },
    {
      "id": "anthropic-claude-connectors",
      "title": "About custom connectors (remote MCP servers) — Claude Help Center",
      "url": "https://support.claude.com/en/articles/11175166-about-custom-connectors-remote-mcp-servers",
      "publisher": "Anthropic",
      "type": "vendor-doc",
      "keyFindings": "Directly contradicts the audit's named consumer. Claude requires manual URL entry, not auto-discovery: users 'Navigate to Customize > Connectors', 'Click + then Add custom connector', and 'Add your connector's remote MCP server URL.' No auto-detection from website HTML or /.well-known/ files is mentioned.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "anthropic-claude-custom-connectors",
      "title": "Third party connectors with remote MCP — Claude Docs",
      "url": "https://claude.com/docs/connectors/custom/remote-mcp",
      "type": "vendor-doc",
      "publisher": "Anthropic",
      "verified": "2026-08-20",
      "keyFindings": "'You can manually add any third-party connector to Claude as long as you have the URL of that remote MCP server.' Steps for every plan tier are: navigate to Connectors → Add custom connector → 'Enter the remote MCP server URL'. Discovery is either the curated Connectors Directory or manual URL entry. NO .well-known lookup, NO domain-derived discovery is documented anywhere on the page."
    },
    {
      "id": "anthropic-claude-for-chrome",
      "title": "Piloting Claude for Chrome",
      "url": "https://claude.com/blog/claude-for-chrome",
      "publisher": "Anthropic",
      "type": "announcement",
      "keyFindings": "Defines instructions hidden in web pages/DOM as prompt injection; describes system-prompt hardening and 'advanced classifiers to detect suspicious instruction patterns' to make Claude NOT follow page-embedded instructions.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "anthropic-claudebot",
      "title": "Does Anthropic crawl data from the web, and how can site owners block the crawler?",
      "url": "https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler",
      "publisher": "Anthropic",
      "type": "vendor-doc",
      "keyFindings": "\"Anthropic's Bots respect 'do not crawl' signals by honoring industry standard directives in robots.txt\" plus non-standard Crawl-delay support and IP JSON lists. No HTTP response headers, no security headers, no security.txt, no page-speed or resource-hint guidance.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "anthropic-computer-use-images",
      "title": "Computer use tool",
      "url": "https://platform.claude.com/docs/en/docs/agents-and-tools/tool-use/computer-use-tool",
      "publisher": "Anthropic",
      "type": "vendor-doc",
      "keyFindings": "Computer use perceives only screenshots and zoom images, with no accessibility tree or DOM access. A second named agent for which ARIA roles on images are entirely inert.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "anthropic-computer-use-tool",
      "title": "Computer use tool",
      "url": "https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool",
      "type": "vendor-doc",
      "publisher": "Anthropic",
      "verified": "2026-08-20",
      "keyFindings": "COUNTER-EVIDENCE for a blanket 'agents read the a11y tree' claim. The desktop computer-use tool is screenshot-only: 17 member actions (screenshot, zoom, clicks, scroll, type, key, wait) with no DOM or accessibility-tree access. Doc redirects web work to the browser use tool: 'For tasks that stay inside webpages, the browser use tool is the closer fit: its member tools read and act on the page itself.'"
    },
    {
      "id": "anthropic-contextual-retrieval",
      "title": "Introducing Contextual Retrieval",
      "url": "https://www.anthropic.com/news/contextual-retrieval",
      "type": "vendor-doc",
      "publisher": "Anthropic",
      "verified": "2026-08-20",
      "keyFindings": "Vendor-documented mechanism for why self-contained passages retrieve better. On a chunk lacking its own context: 'However, this chunk on its own doesn't specify which company it's referring to or the relevant time period, making it difficult to retrieve the right information or use the information effectively.' Measured effect of restoring self-containment by prepending context: contextual embeddings alone reduce top-20 retrieval failure rate 35% (5.7% → 3.7%); with contextual BM25, 49% (→2.9%); with reranking, 67% (→1.9%). Note the fix is applied inside the RAG pipeline, not by the publisher — so this proves the mechanism, not the publisher-side lever."
    },
    {
      "id": "anthropic-crawler",
      "title": "Does Anthropic crawl data from the web, and how can site owners block the crawler?",
      "url": "https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler",
      "publisher": "Anthropic",
      "type": "vendor-doc",
      "keyFindings": "'Anthropic's Bots respect \"do not crawl\" signals by honoring industry standard directives in robots.txt' for ClaudeBot, Claude-User and Claude-SearchBot; examples shown are root-level Disallow: / only, with path granularity not explicitly confirmed.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "anthropic-crawler-docs",
      "title": "Does Anthropic crawl data from the web, and how can site owners block the crawler?",
      "url": "https://support.claude.com/en/articles/8896518",
      "type": "vendor-doc",
      "publisher": "Anthropic",
      "verified": "2026-08-20",
      "keyFindings": "Documents ClaudeBot (training), Claude-User (user-directed fetch), Claude-SearchBot (search indexing). States bots 'respect do not crawl signals by honoring industry standard directives in robots.txt'. COUNTER-EVIDENCE: no mention of sitemaps, RSS, canonical, nofollow, or how URLs are discovered."
    },
    {
      "id": "anthropic-crawler-framework",
      "title": "Does Anthropic crawl data from the web, and how can site owners block the crawler?",
      "url": "https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler",
      "publisher": "Anthropic",
      "type": "vendor-doc",
      "verified": "2026-08-21",
      "keyFindings": "Provides no technical specification about rendering capabilities or content discovery for ClaudeBot, Claude-User or Claude-SearchBot; scope is robots.txt honoring and Crawl-delay support.",
      "accessed": "2026-08-21"
    },
    {
      "id": "anthropic-crawler-support",
      "title": "Does Anthropic crawl data from the web, and how can site owners block the crawler?",
      "url": "https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler",
      "type": "vendor-doc",
      "publisher": "Anthropic",
      "verified": "2026-08-21",
      "keyFindings": "Canonical URL moved from support.anthropic.com to support.claude.com (301). Three documented tokens: ClaudeBot ('collecting web content that could potentially contribute to their training'), Claude-User ('When individuals ask questions to Claude, it may access websites using a Claude-User agent'), Claude-SearchBot ('navigates the web to improve search result quality'). Blanket compliance statement: 'Anthropic's Bots respect do not crawl signals by honoring industry standard directives in robots.txt'. Explicitly names support for the 'Crawl-delay extension to robots.txt'. IP verification list at https://claude.com/crawling/bots.json. NEGATIVE: 'anthropic-ai' and 'Claude-Web' are NOT mentioned anywhere on the current page."
    },
    {
      "id": "anthropic-crawlers",
      "title": "Does Anthropic crawl data from the web, and how can site owners block the crawler?",
      "url": "https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler",
      "type": "vendor-doc",
      "publisher": "Anthropic",
      "verified": "2026-08-21",
      "keyFindings": "Anthropic operates three crawlers: ClaudeBot (collects web content for model training and development), Claude-User (retrieves sites when a user directs Claude to specific content), Claude-SearchBot (indexes web content to improve search result quality). Notable NEGATIVE finding: the document contains no guidance whatsoever on how content is selected, ranked or cited by Claude — no mention of authorship, credentials, publication dates, structured data, brand or source authority. It covers only crawler transparency and robots.txt control. This is direct absence-of-documentation counter-evidence for every authority signal in this domain."
    },
    {
      "id": "anthropic-cu-tool",
      "title": "Computer use tool — security and prompt injection guidance",
      "url": "https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool",
      "type": "vendor-doc",
      "publisher": "Anthropic",
      "verified": "2026-08-20",
      "keyFindings": "'In some circumstances, Claude will follow commands found in content even when they conflict with your instructions. For example, instructions on webpages or contained in images might override your instructions.' Classifiers run on screenshots to flag injections and force user confirmation. Also recommends asking a human to confirm consequential actions — the vendor-side counterpart to the site-side 'don't hide consequence behind a GET link' check."
    },
    {
      "id": "anthropic-define-tools",
      "title": "Anthropic — define tools",
      "url": "https://platform.claude.com/docs/en/agents-and-tools/tool-use/define-tools",
      "type": "vendor-doc",
      "publisher": "Anthropic",
      "verified": "2026-08-21",
      "keyFindings": "A tool `name` \"Must match the regex `^[a-zA-Z0-9_-]{1,64}$`\", so an operationId carrying spaces, punctuation or more than 64 characters cannot be registered verbatim."
    },
    {
      "id": "anthropic-docs-head",
      "title": "Anthropic developer docs page HTML (live head inspection)",
      "url": "https://platform.claude.com/en/docs/overview",
      "publisher": "Anthropic",
      "type": "vendor-doc",
      "keyFindings": "27 <link> tags in head, none referencing llms.txt or llms-full.txt — despite Anthropic serving a 33.5 MB llms-full.txt. Discovery is done via a text pointer inside llms.txt instead.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "anthropic-docs-md",
      "title": "Intro to Claude (markdown alternate of the HTML docs page)",
      "url": "https://platform.claude.com/docs/en/intro.md",
      "type": "vendor-doc",
      "publisher": "Anthropic",
      "verified": "2026-08-20",
      "keyFindings": "Fetching the HTML page URL with .md appended returns clean markdown with YAML front matter (title/url/description) instead of an HTML document — production evidence of the llms.txt .md-alternate convention on a major vendor's docs. Note the alternate embeds MDX components (<Tip>, <Steps>, <CardGroup>), which is a fidelity wrinkle a fidelity check must tolerate."
    },
    {
      "id": "anthropic-llms-txt",
      "title": "Anthropic Developer Documentation llms.txt",
      "url": "https://platform.claude.com/llms.txt",
      "publisher": "Anthropic",
      "type": "vendor-doc",
      "keyFindings": "Spec-compliant llms.txt that ends with 'For more comprehensive documentation, see llms-full.txt' — an AI vendor both publishing and cross-linking the file. The referenced llms-full.txt returns HTTP 200 at 33.5 MB.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "anthropic-tool-use-overview",
      "title": "Anthropic — tool use overview",
      "url": "https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview",
      "type": "vendor-doc",
      "publisher": "Anthropic",
      "verified": "2026-08-21",
      "keyFindings": "\"Claude determines when to call a tool based on the user's request and the tool's description.\""
    },
    {
      "id": "ap2",
      "title": "Agent Payments Protocol (AP2)",
      "url": "https://ap2-protocol.org/",
      "type": "draft-spec",
      "publisher": "Google",
      "verified": "2026-08-20",
      "keyFindings": "Open protocol for the agent economy, positioned as an extension of A2A and the Universal Commerce Protocol. Current terminology is Checkout Mandate (open = user constraints pre-cart; closed = authorization for a finalized checkout) and Payment Mandate (open = payment constraints for autonomous execution; closed = authorization for a specific amount) — note the older 'Intent Mandate'/'Cart Mandate' names are no longer used. Built on Verifiable Digital Credentials (VDCs): tamper-evident cryptographically signed objects forming a non-repudiable audit trail. x402 appears in payment samples. NEGATIVE RESULT: no merchant-side endpoints, agent cards, or well-known URLs are specified on the overview, so AP2 offers no site-auditable surface today."
    },
    {
      "id": "apievangelist-api-catalog-adoption",
      "title": "Only Four API Providers Publish a Real .well-known/api-catalog Right Now",
      "url": "https://apievangelist.com/blog/2026/05/22/four-providers-publishing-well-known-api-catalog/",
      "type": "study",
      "publisher": "API Evangelist",
      "verified": "2026-08-20",
      "keyFindings": "Published 2026-05-22. Methodology: 518 parallel HTTPS requests across 74 providers × 6 host candidates (apex, docs., developer., developers., api., platform.). Result: only FOUR served a valid RFC 9727 Linkset — Cloudflare (1 entry), Memesio (2), Merge.dev (10), Zuplo (1). Two clean 404s; SIXTY-EIGHT returned 200 OK with an HTML body — 'an HTML 200 at a well-known path lies, which is worse than a 404.' Stripe and Twilio have not implemented it."
    },
    {
      "id": "apple-app-search-web-markup",
      "title": "App Search Programming Guide: Mark Up Web Content",
      "url": "https://developer.apple.com/library/archive/documentation/General/Conceptual/AppSearch/WebContent.html",
      "type": "vendor-doc",
      "publisher": "Apple (Documentation Archive)",
      "verified": "2026-08-21",
      "keyFindings": "ARCHIVED documentation. \"To annotate your web content so that users can see rich search results, use standards-based markup for structured data, such as that defined at Schema.org.\" Applebot-supported schemas listed: AggregateRating, Offers, PriceRange, InteractionCount, Organization, Recipe, SearchAction, ImageObject; Open Graph also supported. Ranking claim: \"the popularity of a URL and the amount of structured data available\" contribute to relevancy/ranking in Spotlight/Safari/Siri. The only vendor doc naming SearchAction as consumed — but it is archived and predates Apple Intelligence."
    },
    {
      "id": "apple-applebot-about",
      "title": "About Applebot",
      "url": "https://support.apple.com/en-us/119829",
      "type": "vendor-doc",
      "publisher": "Apple",
      "verified": "2026-08-21",
      "keyFindings": "Page resolves (title: 'About Applebot - Apple Support') and is Apple's canonical Applebot reference documenting Applebot and Applebot-Extended robots.txt tokens. CAVEAT: page body is JS-rendered and could not be extracted verbatim by automated fetch; only the document title was recoverable. Treat quoted specifics as unverified by us."
    },
    {
      "id": "apple-applebot-archived-2025",
      "title": "About Applebot — archived snapshot, 2 March 2025 (Wayback Machine)",
      "url": "https://web.archive.org/web/20250302012726/https://support.apple.com/en-us/119829",
      "type": "vendor-doc",
      "publisher": "Apple / Internet Archive",
      "verified": "2026-08-20",
      "keyFindings": "Verbatim: 'Applebot accesses many kinds of resources from web servers, including but not limited to robots.txt, sitemaps, RSS feeds, HTML, sub resources needed to render pages such as javascript, Ajax requests, images, and more.' This is the strongest single vendor statement that an AI-serving crawler (Applebot powers Siri, Spotlight, Safari and Apple Intelligence grounding) reads sitemaps AND RSS feeds. Removed in the June 2026 revision."
    },
    {
      "id": "apple-applebot-docs",
      "title": "About Applebot",
      "url": "https://support.apple.com/en-us/119829",
      "type": "vendor-doc",
      "publisher": "Apple",
      "verified": "2026-08-21",
      "keyFindings": "Strongest single AI-specific head-tag source. \"Applebot supports robots meta tags in HTML documents.\" Directives: noindex (\"won't appear in Spotlight or Siri Suggestions\"); nosnippet (\"Applebot won't generate a description or web answer for the page. Any suggestions to visit this URL will only include the page's title. Apple will not use data tagged nosnippet as additional context and up-to-date content when AI models are used to generate output for display in Apple products and services\"); nofollow; none; all. Supports the applebot-specific meta name and X-Robots-Tag. Ranking factors listed are engagement, relevancy, links, location, webpage design characteristics — no head-tag factors named."
    },
    {
      "id": "apple-applebot-training-privacy",
      "title": "Applebot model training and individual privacy rights",
      "url": "https://support.apple.com/en-us/120320",
      "type": "vendor-doc",
      "publisher": "Apple",
      "verified": "2026-08-20",
      "keyFindings": "Apple's stated position: web publishers can use standard robots.txt directives to direct Applebot not to crawl, or to direct Apple not to use their content to train Apple's foundation models (Applebot-Extended). Applebot-Extended is a training/AI-use opt-out token only and does not remove a site from Apple search surfaces (Siri/Spotlight/Safari Suggestions). CAVEAT: page body JS-rendered; title-only extraction succeeded."
    },
    {
      "id": "applebot",
      "title": "About Applebot",
      "url": "https://support.apple.com/en-us/119829",
      "publisher": "Apple",
      "type": "vendor-doc",
      "keyFindings": "Explicit path-level examples: 'User-agent: Applebot / Allow: / / Disallow: /private/' and, for the generative-AI training token, 'User-agent: Applebot-Extended / Disallow: /private/'. Confirms AI-relevant crawlers honor per-directory rules.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "applebot-doc",
      "title": "About Applebot",
      "url": "https://support.apple.com/en-us/119829",
      "publisher": "Apple",
      "type": "vendor-doc",
      "keyFindings": "Applebot powers Spotlight, Siri, Safari search and Apple Intelligence. The ONLY schema.org property Apple documents supporting is isAccessibleForFree ('Applebot supports the schema.org isAccessibleForFree property'). No speakable, no Actions, no HowTo. Directly refutes the audit's 'Siri uses speakable' claim.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "ard-conformance-example",
      "title": "ARD conformance example basic/ai-catalog.json",
      "url": "https://raw.githubusercontent.com/ards-project/ard-spec/main/conformance/examples/basic/ai-catalog.json",
      "publisher": "ARD Project",
      "type": "spec",
      "keyFindings": "Top-level keys are specVersion, host, entries. Entries carry identifier, displayName, type (media type), url, description, representativeQueries, capabilities. No `services`, `owner`, `contact`, or `lastUpdated`.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "ard-spec",
      "title": "Agentic Resource Discovery (ARD) Specification",
      "url": "https://agenticresourcediscovery.org/spec/",
      "type": "draft-spec",
      "publisher": "ARDS Project",
      "verified": "2026-08-20",
      "keyFindings": "v0.9 Draft, status 'Proposal', dated 2026-05-28. Manifest ai-catalog.json requires specVersion, host, entries. Each entry requires identifier (urn:air:<publisher>:<namespace>:<agent-name>), displayName, type, and exactly one of url|data. Four discovery mechanisms: /.well-known/ai-catalog.json, robots.txt 'Agentmap:', <link rel=\"ai-catalog\">, DNS SVCB records."
    },
    {
      "id": "ard-spec-discovery",
      "title": "ARD Specification §6.1 — discovery mechanisms",
      "url": "https://raw.githubusercontent.com/ards-project/ard-spec/main/spec/ard.md",
      "publisher": "ARD Project (Google/Microsoft/Hugging Face authors)",
      "type": "spec",
      "keyFindings": "Four advertisement mechanisms: well-known URI; robots.txt 'Agentmap:' entry; '<link rel=\"ai-catalog\" href=\"...\"> in the <head> of a document'; DNS SVCB with TXT fallback (DNS-AID draft).",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "ard-spec-entry-url",
      "title": "ARD Specification — entry schema (url vs data)",
      "url": "https://raw.githubusercontent.com/ards-project/ard-spec/main/spec/ard.md",
      "publisher": "ARD Project",
      "type": "spec",
      "keyFindings": "Each entry carries either a `url` remote reference or embedded `data`; `type` is an IANA media type identifying the artifact. Confirms url is the field to validate, and that entries without url are legal.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "ard-spec-fields",
      "title": "ARD Specification §4.1 manifest fields",
      "url": "https://raw.githubusercontent.com/ards-project/ard-spec/main/spec/ard.md",
      "publisher": "ARD Project (Google/Microsoft/Hugging Face authors)",
      "type": "spec",
      "keyFindings": "Required top-level: specVersion, host, entries. Entry fields: identifier (URN), displayName, type, url|data, description, capabilities, representativeQueries; optional version, updatedAt, tags, metadata, trustManifest. No owner/contact/lastUpdated; no services.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "ard-spec-repo",
      "title": "ards-project/ard-spec (repository)",
      "url": "https://github.com/ards-project/ard-spec",
      "type": "repo",
      "publisher": "ards-project / Linux Foundation",
      "keyFindings": "§4.2 requires exactly one of `url` or `data` per entry, so an entry that embeds its artifact has no endpoint to dereference and is still fully conformant.",
      "verified": "2026-08-24"
    },
    {
      "id": "ardvark-crawler",
      "title": "ardvark — ARD crawler/probe (GitHub code search hits)",
      "url": "https://github.com/HelgeSverre/ardvark",
      "publisher": "HelgeSverre",
      "type": "repo",
      "keyFindings": "Code search shows internal/crawler, internal/probe, internal/fetch and internal/mcpserver all referencing ai-catalog.json — an independent crawler that fetches and probes catalogs and their URLs. (Identified via GitHub code search; repo page not individually fetched.)",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "aria-apg-dialog-modal",
      "title": "ARIA Authoring Practices — modal dialog pattern",
      "url": "https://www.w3.org/WAI/ARIA/apg/patterns/dialog-modal",
      "type": "spec",
      "publisher": "W3C",
      "verified": "2026-08-21",
      "keyFindings": "Requires a dialog to have \"a value set for the aria-labelledby property that refers to a visible dialog title\" or \"a label specified by aria-label\"."
    },
    {
      "id": "aria-in-html-address",
      "title": "ARIA in HTML",
      "url": "https://www.w3.org/TR/html-aria/",
      "publisher": "W3C",
      "type": "spec",
      "keyFindings": "Assigns `address` the implicit `role=group`. This is the only formal machine semantics the element carries, and an unnamed group conveys no contact-information meaning; Chromium in practice exposes it as generic.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "aria-in-html-aside",
      "title": "ARIA in HTML",
      "url": "https://www.w3.org/TR/html-aria/",
      "publisher": "W3C",
      "type": "spec",
      "keyFindings": "Ratified spec assigning `aside` the implicit `role=complementary` (and `address` the implicit `role=group`). Gives <aside> a standardised landmark identity that accessibility-tree-reading agents receive.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "aria-in-html-img",
      "title": "ARIA in HTML",
      "url": "https://www.w3.org/TR/html-aria/",
      "publisher": "W3C",
      "type": "spec",
      "keyFindings": "Ratified spec governing implicit roles and permitted role overrides for HTML elements; confirms the implicit-role machinery by which alt=\"\" already yields presentation/none rather than requiring an author-supplied role.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "arxiv-agent-permissions",
      "title": "How Agents Ask for Permission: User Permissions for AI Agents, from Interfaces to Enforcement",
      "url": "https://arxiv.org/html/2607.13718v2",
      "publisher": "arXiv",
      "type": "study",
      "keyFindings": "Surveys permission models across chatbots, desktop agents and agentic browsers. Finds ChatGPT agent mode operates a remote browser with no user-configurable permissions: \"The agentic browser had no permissions available to grant\" and it acts \"simply by not assigning any permissions policies to the remote browser at all\". Crucially, the paper never mentions the HTTP Permissions-Policy response header or any website-set permission signal — agent permissions are an agent-platform concern, not a site-header concern.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "arxiv-agent-ready-websites",
      "title": "Designing Agent-Ready Websites for AI Web Agents: A Framework for Machine Readability, Actionability, and Decision Reliability",
      "url": "https://arxiv.org/abs/2607.12056",
      "type": "study",
      "publisher": "arXiv",
      "verified": "2026-08-20",
      "keyFindings": "Submitted 2026-07-13. Compared a baseline human-oriented site against an 'agent-ready' variant adding machine readability, semantic clarity, agent actionability, and contextual decision-reliability signals. 300 runs across GPT-4.1, Gemini-2.5 Flash and Grok-4 Fast. Strict success 89.3% (134/150 PASS) vs 49.3% baseline (74/150); partial outcomes fell from 43 to 3; mean steps fell 9.31 → 6.49. Largest gains in product detail extraction, comparison, and multi-constraint selection, attributed to structural clarity, action cues, evidence signals, and temporal validity indicators."
    },
    {
      "id": "arxiv-geo-at-scale-brand-visibility",
      "title": "Generative Engine Optimization at Scale: Measuring Brand Visibility Across AI Search Engines",
      "url": "https://arxiv.org/abs/2606.20065",
      "type": "study",
      "publisher": "arXiv",
      "verified": "2026-08-20",
      "keyFindings": "Submitted 2026-06-18. 100,000+ prompt responses across 100+ brands, March–May 2026, via the Ranqo platform, covering ChatGPT, Claude, Perplexity and Gemini. Three-tier brand-stature ladder: global brands appear in 73% of relevant answers, mid-market 44%, niche 11% (~30pp per tier). Citation destinations: 78% to corporate websites; then YouTube, Reddit, editorial media, Wikipedia. 'Best-of' listicles are the highest-leverage format at ~21% of all citations. Sentiment framing flips ~6.7x more often than mention presence."
    },
    {
      "id": "arxiv-geo-critical-survey",
      "title": "Optimizing Visibility in Generative Engines: A Critical Survey of Generative Engine Optimization (2023-2026)",
      "url": "https://arxiv.org/abs/2607.14035",
      "type": "study",
      "publisher": "arXiv",
      "verified": "2026-08-20",
      "keyFindings": "Submitted 2026-07-15. Survey of 45 studies. Headline negative conclusion: 'no reviewed technique shows a stable, longitudinal, cross-platform causal effect on organic discoverability or downstream behavior.' The original GEO paper's gains are 'conditional on a source already being present in a fixed context; they establish neither organic discoverability nor durable traffic effects.' Generic heuristics transfer poorly; competition erodes individual gains. Commercial audits show low source overlap and substantial run-to-run variability. Only reproducible lever: already-retrieved content can causally alter its own citation or use. Essential counter-evidence for any scored GEO claim."
    },
    {
      "id": "arxiv-geo-flag-prevalence",
      "title": "GEO-Flag: Detecting and Measuring GEO-Optimized Web Content",
      "url": "https://arxiv.org/abs/2608.16824",
      "type": "study",
      "publisher": "arXiv",
      "verified": "2026-08-20",
      "keyFindings": "Submitted 2026-08-17. Introduces GEOFlagBench (3,200 webpages across 400 queries) and Intervention-Paired Training on ModernBERT (F1 0.862 → 0.944). Field measurement over real Google Search and Gemini-grounded results for 1,000 real queries: ~8.90% of available pages show GEO optimization, rising to 16.36% among pages modified in 2026. First credible prevalence estimate for GEO-optimized content in the wild."
    },
    {
      "id": "arxiv-hotel-algorithm-audit",
      "title": "Whose hotel does the AI recommend? An algorithm audit of reputation signals in LLM-assisted hotel selection",
      "url": "https://arxiv.org/abs/2606.16344",
      "type": "study",
      "publisher": "arXiv",
      "verified": "2026-08-20",
      "keyFindings": "Submitted 2026-06-15. Twelve LLMs, randomized scenarios independently varying rating, review volume, recency, management responses, chain status, price, eco-certification, and list position. Top rating raises selection probability by 31.6pp; high price lowers it by 30pp. Models overweight eco-certification versus human behavior and largely ignore management responses. List position — a content-free artifact — causally shifts recommendations by roughly $12/night in equivalent value. Stated reasons track revealed weights imperfectly."
    },
    {
      "id": "arxiv-structured-linked-data-memory-layer",
      "title": "Structured Linked Data as a Memory Layer for Agent-Orchestrated Retrieval",
      "url": "https://arxiv.org/abs/2603.10700",
      "type": "study",
      "publisher": "arXiv (Volpini, Raad, Gamba, Riccitelli)",
      "verified": "2026-08-20",
      "keyFindings": "Submitted 11 Mar 2026. Compares plain HTML vs HTML+JSON-LD vs agent-optimized entity pages across four domains using Google Vector Search + ADK. Finding: \"JSON-LD markup alone provides only modest improvements\" to standard RAG; enhanced entity pages (agent instructions + neural search) gave +29.6% accuracy for standard RAG and +29.8% for the full agentic pipeline. Supports: markup alone ≠ retrieval win; the win comes from purpose-built machine-readable entity surfaces."
    },
    {
      "id": "arxiv-what-gets-cited-geo",
      "title": "What Gets Cited: Competitive GEO in AI Answer Engines",
      "url": "https://arxiv.org/abs/2605.25517",
      "type": "study",
      "publisher": "arXiv",
      "verified": "2026-08-20",
      "keyFindings": "Submitted 2026-05-25. Controlled two-document RAG testbed injecting exactly two candidate sources; measures which receives the first citation marker. 252,000 trials, six LLMs, factorial design over 18 content factors varying one element at a time. Ranked drivers: topical relevance, list position (first-source advantage, ~12 citations per 100), explicit price information, recent timestamp. Completeness and trust signals gave modest gains. NULL FINDING: formatting-only edits have little impact. Ships an evaluation protocol and GEO checklist."
    },
    {
      "id": "autogeo-arxiv",
      "title": "What Generative Search Engines Like and How to Optimize Web Content Cooperatively (AutoGEO)",
      "url": "https://arxiv.org/abs/2510.11438",
      "type": "study",
      "publisher": "arXiv (Wu, Zhong, Kim, Xiong), submitted 2025-10-13",
      "verified": "2026-08-20",
      "keyFindings": "Extracts generative-engine 'preference rules' from LLM explanations and applies them to rewrite content, evaluated against Google AI Overview and ChatGPT plus datasets built from real user queries. The second paper competitors cite (geo-optimizer-skill labels it 'ICLR 2026'; the arXiv listing shows cs.IR preprint status, so the venue claim is a competitor assertion, not something the arXiv page confirms)."
    },
    {
      "id": "awesome-open-geo",
      "title": "Awesome Open GEO",
      "url": "https://github.com/cloudapp-dev/awesome-open-geo",
      "type": "article",
      "publisher": "cloudapp-dev (CC0-1.0, 2 stars, created 2026-07-29)",
      "verified": "2026-08-20",
      "keyFindings": "Curated OSI-licensed-only landscape covering Standards, Audit and Linting, Visibility Monitoring, llms.txt tooling, Framework integrations, Crawler access/log analysis, Agent skills/MCP, Research, and Datasets. Its framing states the problem we exploit: 'The tooling around it, however, is almost entirely paid SaaS ... every entry is free to use and published under an OSI-approved license, so you can read the source, self-host it, and audit what it measures.' Useful as a map of every OSS competitor in one verified place."
    },
    {
      "id": "ax-score-repo",
      "title": "AX Score — The Lighthouse for AI Agents",
      "url": "https://github.com/agentgram/ax-score",
      "type": "article",
      "publisher": "AgentGram — 3 stars, MIT, pushed 2026-08-20",
      "verified": "2026-08-20",
      "keyFindings": "Direct name-space competitor. npm CLI running 19 audits across six categories (Discovery, API Quality, Structured Data, Auth & Onboarding, Error Handling, Documentation) producing an overall 0-100. Notably ships a --repeat flag reporting score stability (mean, range, delta, variance across runs) — the only competitor addressing run-to-run variability, which Lighthouse documents in docs/variability.md. Uploads results to a hosted AgentGram API. No published evidence per audit, no weights, no consumer mapping; recommendations such as 'Implement Ed25519 cryptographic signatures' are asserted with no cited consumer."
    },
    {
      "id": "axe-aria-dialog-name",
      "title": "axe rule — aria-dialog-name",
      "url": "https://dequeuniversity.com/rules/axe/4.10/aria-dialog-name",
      "type": "vendor-doc",
      "publisher": "Deque",
      "verified": "2026-08-21",
      "keyFindings": "\"Screen reader users are not able to discern the purpose of elements with `role=\"dialog\"` or `role=\"alertdialog\"` that do not have an accessible name\" (impact: serious; Deque best practice, not a WCAG success criterion)."
    },
    {
      "id": "axe-aria-hidden-body",
      "title": "axe rule — aria-hidden-body",
      "url": "https://dequeuniversity.com/rules/axe/4.10/aria-hidden-body",
      "type": "vendor-doc",
      "publisher": "Deque",
      "verified": "2026-08-21",
      "keyFindings": "\"Document content is not accessible to assistive technology if `<body aria-hidden=\"true\">`\" (impact: critical)."
    },
    {
      "id": "axe-aria-valid-attr-value",
      "title": "axe rule — aria-valid-attr-value",
      "url": "https://dequeuniversity.com/rules/axe/4.10/aria-valid-attr-value",
      "type": "vendor-doc",
      "publisher": "Deque",
      "verified": "2026-08-21",
      "keyFindings": "ARIA values \"must be spelled correctly and correspond to values that make sense for a particular attribute in order to perform the intended accessibility function\"; a checkbox role becomes non-functional given an invalid value."
    },
    {
      "id": "axe-core-rule-descriptions",
      "title": "axe-core Rule Descriptions",
      "url": "https://github.com/dequelabs/axe-core/blob/develop/doc/rule-descriptions.md",
      "type": "vendor-doc",
      "publisher": "Deque Labs (axe-core, MPL-2.0, ~7.4k stars)",
      "verified": "2026-08-20",
      "keyFindings": "Every rule carries a fixed field set: Rule ID, Description, Impact (Critical/Serious/Moderate/Minor), Tags (WCAG version + level + standard), Issue Type (failure vs needs-review), and ACT Rules mapping. Rules are partitioned by conformance tier, and — most relevant to us — there are explicit maturity tiers: 'Experimental' rules are 'disabled by default in axe-core and the axe browser extensions' and tagged experimental; 'Deprecated' rules are 'disabled by default and will be removed in the next major release'; WCAG 2.2 rules were 'disabled by default, until WCAG 2.2 is more widely adopted'. This is a direct precedent for our scored / informative / experimental tiering and for gating scoring on adoption maturity."
    },
    {
      "id": "axe-duplicate-id-aria",
      "title": "axe rule — duplicate-id-aria",
      "url": "https://dequeuniversity.com/rules/axe/4.10/duplicate-id-aria",
      "type": "vendor-doc",
      "publisher": "Deque",
      "verified": "2026-08-21",
      "keyFindings": "An id \"must be unique to prevent the second instance from being overlooked by assistive technology\"; \"only the first instance gets acted upon by client-side scripting\" (impact: critical)."
    },
    {
      "id": "axe-frame-title",
      "title": "axe rule — frame-title",
      "url": "https://dequeuniversity.com/rules/axe/4.10/frame-title",
      "type": "vendor-doc",
      "publisher": "Deque",
      "verified": "2026-08-21",
      "keyFindings": "Maps to WCAG 4.1.2 Name, Role, Value (Level A) with impact \"serious\"; the stated consumer is screen reader users, who \"can access a list of all frame titles on a page\"."
    },
    {
      "id": "bing-chat-controls-2023",
      "title": "Announcing new options for webmasters to control usage of their content in Bing Chat",
      "url": "https://blogs.bing.com/webmaster/september-2023/Announcing-new-options-for-webmasters-to-control-usage-of-their-content-in-Bing-Chat",
      "type": "vendor-doc",
      "publisher": "Microsoft Bing Webmaster Blog (22 Sept 2023)",
      "verified": "2026-08-20",
      "keyFindings": "Head-tag directives with explicitly documented AI-answer effects. \"Content without NOCACHE tag and without NOARCHIVE tag may be included in Bing Chat answers.\" \"Content with the NOCACHE tag may be included in Bing Chat answers. We will only display URL/Snippet/Title in the answer.\" \"Content tagged NOARCHIVE will not be included in Bing Chat answers, not be linked to in the answers.\" Both tags together are treated as NOCACHE; tagged content still appears in normal Bing results."
    },
    {
      "id": "bing-indexnow",
      "title": "IndexNow: Instantly Index your Web Content in Search Engines",
      "url": "https://blogs.bing.com/webmaster/october-2021/IndexNow-Instantly-Index-your-web-content-in-Search-Engines",
      "type": "vendor-doc",
      "publisher": "Microsoft Bing Webmaster Blog",
      "verified": "2026-08-20",
      "keyFindings": "Confirms the key-file-at-root verification flow and the motivation (organic discovery 'can take days or even weeks'). No published crawl-latency SLA."
    },
    {
      "id": "bing-sitemaps-ai-search",
      "title": "Keeping Content Discoverable with Sitemaps in AI-Powered Search (Bing Webmaster Blog, July 2025)",
      "url": "https://blogs.bing.com/webmaster/July-2025/Keeping-Content-Discoverable-with-Sitemaps-in-AI-Powered-Search",
      "type": "vendor-doc",
      "publisher": "Microsoft Bing",
      "verified": "2026-08-21",
      "keyFindings": "The strongest CURRENT vendor statement tying sitemaps to AI answers: 'freshness signals directly influence how quickly updates are reflected in search results and AI generated answers'; 'The lastmod field in your sitemap remains a key signal, helping Bing prioritize URLs for recrawling and reindexing'; sitemaps deliver 'comprehensive URL coverage' so 'all your relevant URLs are discoverable — even at enterprise scale'; recommends pairing with IndexNow for real-time submission; ties accurate sitemap signals to 'both traditional search and AI-powered experiences like Copilot'."
    },
    {
      "id": "bing-snippet-controls-2020",
      "title": "Announcing new options for webmasters to control their snippets at Bing",
      "url": "https://blogs.bing.com/webmaster/april-2020/Announcing-new-options-for-webmasters-to-control-their-snippets-at-Bing",
      "type": "vendor-doc",
      "publisher": "Microsoft Bing Webmaster Blog (8 Apr 2020)",
      "verified": "2026-08-20",
      "keyFindings": "Bing announced support for max-snippet:[number], max-image-preview:[value], max-video-preview:[number], alongside existing NOSNIPPET. Establishes that Bing (which grounds Copilot) parses the same robots meta vocabulary as Google."
    },
    {
      "id": "bing-webmaster-guidelines",
      "title": "Bing Webmaster Guidelines",
      "url": "https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a",
      "type": "vendor-doc",
      "publisher": "Microsoft Bing",
      "verified": "2026-08-20",
      "keyFindings": "Canonical URL for the GEO/grounding guidance summarized by sej-bing-geo-guidelines. The URL resolves (HTTP 200) but the page body is client-side rendered and returned no extractable text to automated fetch, so the quotes above could not be verified against the primary document. Marked unverified deliberately; treat the SEJ report as the citable secondary until a rendered fetch succeeds."
    },
    {
      "id": "brave-comet",
      "title": "Comet Prompt Injection: Agentic Browser Security",
      "url": "https://brave.com/blog/comet-prompt-injection/",
      "type": "article",
      "publisher": "Brave Software",
      "verified": "2026-08-20",
      "keyFindings": "Perplexity Comet fed page content to its LLM without separating user instructions from page data. Injection was hidden in a Reddit comment behind a spoiler tag; Brave explicitly names 'white text on white backgrounds, HTML comments, or other invisible elements' as the hiding techniques. PoC chain: agent read hidden instructions from UGC, pulled the user's email from their Perplexity account, triggered an OTP, read the OTP from the already-logged-in Gmail tab, and posted both back to Reddit. Establishes UGC on a third-party site as a live injection surface."
    },
    {
      "id": "brave-search-crawler",
      "title": "Brave Search crawler",
      "url": "https://search.brave.com/help/brave-search-crawler",
      "type": "vendor-doc",
      "publisher": "Brave Software",
      "verified": "2026-08-20",
      "keyFindings": "MAJOR COUNTER-EVIDENCE for the 'BraveBot' token: 'The Brave Search crawler does not advertise a differentiated user agent because we must avoid discrimination from websites that allow only Google to crawl them.' Also states 'robots.txt is not used to prevent a page from being indexed. A site owner can delist a page by using the robots noindex directive.' No mention of Brave Leo / AI training controls tied to a crawler token. This directly undercuts advice to add a 'Bravebot' disallow rule."
    },
    {
      "id": "brave-unseeable",
      "title": "Unseeable Prompt Injections in Screenshots (Comet, Fellou, Opera Neon)",
      "url": "https://brave.com/blog/unseeable-prompt-injections/",
      "type": "article",
      "publisher": "Brave Software",
      "verified": "2026-08-20",
      "keyFindings": "Instructions rendered as faint light-blue text on a yellow background are invisible to humans but recovered by the agent's vision/OCR path. Confirms Opera Neon was exploitable via 'hidden HTML elements and other non-rendered markup' — direct evidence that non-rendered markup (comments, templates, display:none) is an active ingestion channel, not a theoretical one."
    },
    {
      "id": "browser-use-clickable-elements",
      "title": "browser-use ClickableElementDetector source",
      "url": "https://raw.githubusercontent.com/browser-use/browser-use/main/browser_use/dom/serializer/clickable_elements.py",
      "type": "vendor-doc",
      "publisher": "browser-use",
      "verified": "2026-08-20",
      "keyFindings": "The most-used open-source browser-agent library detects interactivity from accessibility semantics: interactive tags (button, input, select, textarea, a, details, summary, option, optgroup); ARIA roles (button, link, menuitem, option, radio, checkbox, tab, textbox, combobox, slider, spinbutton, search, searchbox, row, cell, gridcell); attributes role and tabindex; ARIA state (checked, expanded, pressed, selected); and accessibility properties 'focusable, editable, settable'. Elements lacking these fall back to cursor:pointer and CDP click-listener sniffing."
    },
    {
      "id": "browserarena-arxiv",
      "title": "BrowserArena: Evaluating LLM Agents on Real-World Web Navigation Tasks",
      "url": "https://arxiv.org/abs/2510.02418",
      "type": "study",
      "publisher": "arXiv",
      "verified": "2026-08-20",
      "keyFindings": "Live open-web arena with step-level human feedback. Identifies three consistent failure categories for browsing agents: CAPTCHA resolution, pop-up banner removal, and direct URL navigation. Concludes findings 'surface both the diversity and brittleness of current web agents' — i.e. agents fail on live-web friction, not on missing metadata."
    },
    {
      "id": "byte5ai-agent-readiness-skill",
      "title": "Claude Agent Readiness Skill (GEO audit skill, evidence-tagged)",
      "url": "https://github.com/byte5ai/claude-agent-readiness-skill",
      "type": "article",
      "publisher": "byte5 — 1 star, MIT, created 2026-05-16, last pushed 2026-05-17",
      "verified": "2026-08-20",
      "keyFindings": "The single closest competitor to our differentiation. It already grades evidence: every recommendation carries a tier tag S (platform primary sources) / A (peer-reviewed academic) / B (empirical practitioner studies) plus a confidence flag (method-sound multi-tested non-vendor-interest, solid, plausible-but-limited, or marketing-claim/no-primary-source). It mandates a 'What to skip' section per report naming overhyped practices — explicitly Schema.org for AI citations, '/llms.txt as a visibility lever', blanket date-bumping, programmatic SEO — 'with the data that disproves them'. Knowledge base spans Princeton KDD 2024, Toronto 2025, MIT/Columbia E-GEO, EMNLP 2025, vendor docs (OpenAI, Anthropic, Google, Bing, Vercel, Cloudflare) and an Ahrefs difference-in-differences study (n=1,885 + 4,000 matched controls) reported as showing no measurable Schema effect on AI citation rate. Structural weaknesses we can beat: it is an LLM skill, not a deterministic scanner (no reproducible score, no CI exit code); ~50 check codes live in prose in assets/audit-checks.md; evidence tiers are per-recommendation prose, not per-audit dossiers with resolvable verified URLs; 1 star and untouched since May 2026."
    },
    {
      "id": "cf-ai-crawl-control",
      "title": "Cloudflare AI Crawl Control",
      "url": "https://developers.cloudflare.com/ai-crawl-control/",
      "type": "vendor-doc",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "Documents crawler monitoring, allow/block rules, robots.txt compliance tracking and pay-per-crawl monetization (private beta). NEGATIVE RESULT worth recording: this overview documents no markdown transformation for AI crawlers and no x-markdown-tokens response header — I could not verify that header anywhere, so no proposal below depends on it. The verified token-accounting artifact from Cloudflare is the toMarkdown `tokens` response field instead."
    },
    {
      "id": "cf-content-signals",
      "title": "Cloudflare — Content Signals Policy",
      "url": "https://blog.cloudflare.com/content-signals-policy/",
      "type": "vendor-doc",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "Launched 2025-09-24. robots.txt line 'Content-Signal: search=yes, ai-train=no' scoped to a User-Agent group. Three signals: search, ai-input, ai-train; values yes|no; omission = no preference expressed. Signals govern USE after access, orthogonal to Allow/Disallow which govern ACCESS. Auto-deployed to 3.8M+ domains via Cloudflare's managed robots.txt."
    },
    {
      "id": "cf-tomarkdown-formats",
      "title": "Markdown Conversion — supported formats",
      "url": "https://developers.cloudflare.com/workers-ai/features/markdown-conversion/supported-formats/",
      "type": "vendor-doc",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "text/html (.html/.htm) is an accepted input MIME type alongside PDF, images, Office and CSV. Conversion methodology is not documented, i.e. no guarantee that data: URIs, hidden DOM or JSON script blocks are stripped before tokenization."
    },
    {
      "id": "cf-tomarkdown-rest",
      "title": "Markdown Conversion — REST API usage (Workers AI)",
      "url": "https://developers.cloudflare.com/workers-ai/features/markdown-conversion/usage/rest-api/",
      "type": "vendor-doc",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "The toMarkdown REST response returns fields id, name, mimeType, format, tokens, data — e.g. \"tokens\": 49 for a converted HTML file. A major infra vendor bills/reports HTML→markdown conversion in tokens per document, so per-page token count is a first-class, vendor-visible unit."
    },
    {
      "id": "checkly-content-negotiation-state",
      "title": "The Current State of Content Negotiation for AI Agents (Feb 2026)",
      "url": "https://www.checklyhq.com/blog/state-of-ai-agent-content-negotation/",
      "type": "study",
      "publisher": "Checkly",
      "verified": "2026-08-20",
      "keyFindings": "Published 2026-02-19. Reproducible method: point each agent at httpbin.org/headers, which echoes request headers. SENDING text/markdown: Claude Code v2.1.38 ('Accept: text/markdown, text/html, */*'), Cursor 2.4.28, OpenCode 1.2.5 (q=1.0 markdown). NOT sending: OpenAI Codex, Gemini CLI, GitHub Copilot, Windsurf. Does not observe any agent using rel=alternate link discovery. Useful as an earlier time-slice showing agent markdown support growing between Feb and Jun 2026."
    },
    {
      "id": "chrome-agentic-forms-guide",
      "title": "Chrome modern-web-guidance: guides/webmcp/agentic-forms",
      "url": "https://raw.githubusercontent.com/GoogleChrome/modern-web-guidance-src/main/guides/webmcp/agentic-forms/guide.md",
      "publisher": "Google Chrome (GoogleChrome/modern-web-guidance-src)",
      "type": "vendor-doc",
      "keyFindings": "web-feature-ids: declarative-webmcp. \"The Declarative API transforms standard HTML <form> elements into WebMCP tools via attributes. The browser synthesizes a JSON Schema from the form inputs and handles agent interactions.\" Documents attribute resolution order (toolparamdescription > <label> textContent > aria-description) and fieldset grouping, plus SubmitEvent.agentInvoked / respondWith().",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "chrome-devtools-mcp-tool-reference",
      "title": "chrome-devtools-mcp tool reference (take_snapshot)",
      "url": "https://raw.githubusercontent.com/ChromeDevTools/chrome-devtools-mcp/main/docs/tool-reference.md",
      "type": "vendor-doc",
      "publisher": "Google / Chrome DevTools",
      "verified": "2026-08-20",
      "keyFindings": "Google's own MCP server: 'Take a text snapshot of the currently selected page based on the a11y tree. The snapshot lists page elements along with a unique identifier (uid).' Confirms a third independent major-vendor agent harness built on the accessibility tree."
    },
    {
      "id": "chrome-devtools-mcp-tools",
      "title": "Chrome DevTools MCP — tool reference",
      "url": "https://github.com/ChromeDevTools/chrome-devtools-mcp/blob/main/docs/tool-reference.md",
      "type": "repo",
      "publisher": "Google / Chrome DevTools",
      "verified": "2026-08-21",
      "keyFindings": "`take_snapshot` is \"based on the a11y tree\", and the click and fill tools consume uids taken from it."
    },
    {
      "id": "chrome-docs-agentic",
      "title": "Lighthouse agentic browsing docs — llms.txt audit",
      "url": "https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt",
      "type": "vendor-doc",
      "publisher": "Google / Chrome Developers",
      "verified": "2026-08-21",
      "keyFindings": "Confirms the shipped docs set: Scoring, Registered WebMCP tools, Forms missing declarative WebMCP, WebMCP schema validity, Accessibility for agents, Layout stability. Explicitly: a 404 on llms.txt is notApplicable because 'providing the file is optional at the moment' — Lighthouse never penalises a missing llms.txt."
    },
    {
      "id": "chrome-docs-declarative-api",
      "title": "WebMCP Declarative API",
      "url": "https://developer.chrome.com/docs/ai/webmcp/declarative-api",
      "publisher": "Google Chrome Developers",
      "type": "vendor-doc",
      "keyFindings": "Documents toolname (\"Clearly name the tool, based on its purpose\"), tooldescription, toolparamdescription (\"Map elements to a property description within the JSON Schema\") and toolautosubmit, with a supportRequestTool form example. Published 2026-05-18, under origin trial.",
      "verified": "2026-08-24",
      "accessed": "2026-08-21"
    },
    {
      "id": "chrome-guidance-corpus",
      "title": "GoogleChrome/modern-web-guidance-src — guides index",
      "url": "https://github.com/GoogleChrome/modern-web-guidance-src/tree/main/guides",
      "publisher": "Google Chrome",
      "type": "repo",
      "keyFindings": "Google's published guidance corpus (accessibility, forms, html, js, ui-behaviors, ui-components, webmcp, ...) routes all agent-affordance guidance through guides/webmcp (agentic-forms, agentic-javascript-tools, webmcp). Repo-scoped code search for `data-action` returns 1 incidental hit and no guide — Google does not advise or consume data-action for agent readability.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "chrome-lighthouse-agentic-browsing",
      "title": "Agentic Browsing category | Lighthouse | Chrome for Developers",
      "url": "https://developer.chrome.com/docs/lighthouse/agentic-browsing",
      "type": "vendor-doc",
      "publisher": "Google Chrome",
      "verified": "2026-08-20",
      "keyFindings": "Category index (Lighthouse 13.3+). Seven documented audits: Scoring, Registered WebMCP tools, Forms missing declarative WebMCP, WebMCP schema validity, llms.txt, Accessibility for agents, Layout stability. There is NO auto-discovery-links / markdown-alternate audit (I fetched that hypothesized URL and got HTTP 404). Category reports a fractional pass ratio, not a 0-100 score."
    },
    {
      "id": "chrome-lighthouse-llms-txt-doc",
      "title": "llms.txt | Lighthouse | Chrome for Developers",
      "url": "https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt",
      "type": "vendor-doc",
      "publisher": "Google Chrome",
      "verified": "2026-08-21",
      "keyFindings": "Official audit doc. Calls llms.txt 'an emerging convention' for 'a machine-readable summary of a website's content, specifically designed for LLMs and AI agents.' Confirms N/A on 404 because 'providing the file is optional at the moment.' Names NO agents that consume it. No mention of llms-full.txt or link tags."
    },
    {
      "id": "chrome-ot-blog",
      "title": "WebMCP origin trial",
      "url": "https://developer.chrome.com/blog/ai-webmcp-origin-trial",
      "publisher": "Google Chrome Developers",
      "type": "announcement",
      "keyFindings": "\"In Chrome 149, you can sign up for the WebMCP origin trial.\" Time-limited early-access program.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "chrome-referrer-default",
      "title": "A new default Referrer-Policy for Chrome: strict-origin-when-cross-origin",
      "url": "https://developer.chrome.com/blog/referrer-policy-new-chrome-default",
      "publisher": "Google Chrome Developers",
      "type": "vendor-doc",
      "keyFindings": "Chrome switched its default referrer policy to strict-origin-when-cross-origin as of version 85. Sites without the header already get exactly the value this audit recommends. \"This is the new default, but websites can still pick a policy of their choice.\"",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "chrome-webmcp-docs",
      "title": "WebMCP — AI on Chrome documentation",
      "url": "https://developer.chrome.com/docs/ai/webmcp",
      "type": "vendor-doc",
      "publisher": "Google Chrome",
      "verified": "2026-08-20",
      "keyFindings": "Documents an imperative JS API and a declarative HTML-forms annotation API; 'WebMCP is available as a Chrome flag for local development' via chrome://flags/#enable-webmcp-testing; 'Join the WebMCP origin trial from Chrome 149.' COUNTER-EVIDENCE: the page names NO production AI agent that currently calls WebMCP tools — only the Model Context Tool Inspector extension for manual developer testing."
    },
    {
      "id": "chrome-webmcp-guide-clientside",
      "title": "Chrome modern-web-guidance: guides/webmcp/webmcp",
      "url": "https://raw.githubusercontent.com/GoogleChrome/modern-web-guidance-src/main/guides/webmcp/webmcp/guide.md",
      "publisher": "Google Chrome (GoogleChrome/modern-web-guidance-src)",
      "type": "vendor-doc",
      "keyFindings": "\"WebMCP runs entirely client-side in the browser tab. It is not a backend server, and it does not use HTTP, Server-Sent Events (SSE), or stdio transports. The web page itself acts as the tool registry.\" Best practices cover naming, schema design, reliability, and \"Tools should be atomic, composable, and distinct\" / \"Register/unregister tools dynamically depending on the current page context\" — no commerce-category coverage requirement anywhere. Also: WebMCP supports Tools only, not Resources or Prompts.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "chrome-webmcp-origin-trial",
      "title": "Join the WebMCP origin trial — Chrome for Developers",
      "url": "https://developer.chrome.com/blog/ai-webmcp-origin-trial",
      "type": "vendor-doc",
      "publisher": "Google Chrome",
      "verified": "2026-08-20",
      "keyFindings": "Published 2026-06-09: 'In Chrome 149, you can sign up for the WebMCP origin trial.' Origin trials described as 'time-limited programs' with 'possible usage limits'. The post does NOT name which agents consume registered tools."
    },
    {
      "id": "chromestatus-webmcp",
      "title": "Chrome Platform Status — WebMCP (feature 5117755740913664)",
      "url": "https://chromestatus.com/feature/5117755740913664",
      "type": "vendor-doc",
      "publisher": "Google Chrome",
      "verified": "2026-08-20",
      "keyFindings": "web_feature id `document-modelcontext`. Dev trial from desktop M146; Origin Trial M149→M156 (origin_trial_id 4163014905550602241). Chrome shipping status: 'Proposed' (NOT shipped to stable). Spec maturity: 'Specification being incubated in a Community Group' (Incubation). Firefox signal: 'No signal'. Safari signal: 'No signal'. Blink component Blink>Agentic Platform>WebMCP. 174 stars."
    },
    {
      "id": "claude-in-chrome-docs",
      "title": "Getting started with Claude in Chrome",
      "url": "https://support.claude.com/en/articles/12012173-getting-started-with-claude-in-chrome",
      "publisher": "Anthropic",
      "type": "vendor-doc",
      "keyFindings": "Describes Claude reading text on webpages and browser console output. Documents no handling of custom HTML data attributes and no data-action convention.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "cloudflare-ai-crawl-control",
      "title": "Cloudflare AI Crawl Control (formerly AI Audit)",
      "url": "https://developers.cloudflare.com/ai-crawl-control/",
      "type": "vendor-doc",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "Adjacent infrastructure-side tool: identifies which AI services access a site, their request patterns, and robots.txt compliance ('track which crawlers follow your directives'), with allow/block rules per crawler across all plans. It is ground truth for whether a bot actually fetched a resource — the observational data no site-scanning audit tool (including ours) can produce from a single crawl. Positions us as complementary, not competing, and gives us a citable source for real-crawler behaviour claims."
    },
    {
      "id": "cloudflare-ai-crawl-control-docs",
      "title": "AI Crawl Control overview",
      "url": "https://developers.cloudflare.com/ai-crawl-control/",
      "type": "vendor-doc",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "Formerly 'AI Audit'. Provides per-crawler allow/block policies, robots.txt compliance tracking and enforcement rules, and pay-per-crawl. Confirms edge bot-management is now a first-class gate between AI crawlers and site content."
    },
    {
      "id": "cloudflare-ai-crawler-purpose",
      "title": "A deeper look at AI crawlers: breaking down traffic by purpose and industry",
      "url": "https://blog.cloudflare.com/ai-crawler-traffic-by-purpose-and-industry/",
      "type": "dataset",
      "publisher": "Cloudflare Radar",
      "verified": "2026-08-20",
      "keyFindings": "Network-scale empirical bot activity. 'Training traffic, responsible for nearly 80% of the crawling from AI bots'; user-action and undeclared purposes together under 5%. Aug 1-7 2025 default view: ClaudeBot and GPTBot 'account for nearly half of the observed crawling activity'; Meta-ExternalAgent in top five; Amazonbot and ByteSpider top-five in Computer & Electronics. July 2025 user-action view: ChatGPT-User 'nearly three quarters of the request traffic', Perplexity-User second. Crawl-to-refer ratios (Aug 1-7 2025): Anthropic ~50,000:1, OpenAI 887:1, Perplexity 118:1 overall; News & Publications 2,500:1 / 152:1 / 32.7:1."
    },
    {
      "id": "cloudflare-ai-crawler-purpose-industry",
      "title": "A deeper look at AI crawlers: breaking down traffic by purpose and industry",
      "url": "https://blog.cloudflare.com/ai-crawler-traffic-by-purpose-and-industry/",
      "type": "article",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "Published 2025-08-28. Training accounts for ~80% of AI bot crawling activity. Defines four crawl-purpose categories: Training (erratic, non-cyclical), Search (index building for AI search / AI Overviews), User Action (query-triggered fetches, shows daily human usage cycles), Undeclared. Crawl-to-refer ratios: Anthropic ClaudeBot ~50,000:1 across all industries; within News & Publications, Anthropic ~2,500:1 and OpenAI ~152:1. Methodology: crawler HTML requests compared against referred human traffic across Cloudflare's customer base."
    },
    {
      "id": "cloudflare-ai-options-2026",
      "title": "Your site, your rules: new AI traffic options for all customers",
      "url": "https://blog.cloudflare.com/content-independence-day-ai-options/",
      "type": "article",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "Announces new defaults effective 15 September 2026: for new Cloudflare domains, 'Training' and 'Agent' class bots are blocked by default on ad-displaying pages while 'Search' remains allowed. Extends managed robots.txt with a `use` signal (immediate / reference / full). Direct evidence that the default posture of a large share of the web is moving toward blocking agents."
    },
    {
      "id": "cloudflare-block-ai-bots",
      "title": "Cloudflare — block AI bots",
      "url": "https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots",
      "type": "vendor-doc",
      "publisher": "Cloudflare",
      "verified": "2026-08-21",
      "keyFindings": "Groups AI bot controls as Search, Agent and Training categories, making the training/realtime split an operational reality rather than a taxonomy invented for this audit."
    },
    {
      "id": "cloudflare-content-independence-day",
      "title": "Content Independence Day: no AI crawl without compensation",
      "url": "https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/",
      "type": "article",
      "publisher": "Cloudflare",
      "verified": "2026-08-21",
      "keyFindings": "Published 2025-07-01. Cloudflare switched to blocking AI crawlers by default for new domains and launched pay-per-crawl. States it is ~750x harder to earn a visitor from OpenAI and ~30,000x harder from Anthropic than under Google's classic crawl-for-traffic bargain. Marks the structural break in the crawl-for-traffic exchange that all crawl-to-refer numbers describe."
    },
    {
      "id": "cloudflare-content-signals",
      "title": "Giving users choice with Cloudflare's new Content Signals Policy",
      "url": "https://blog.cloudflare.com/content-signals-policy/",
      "publisher": "Cloudflare",
      "type": "announcement",
      "keyFindings": "Real-world AI usage-preference signalling (search, ai-input, ai-train) is expressed in robots.txt via 'Content-Signal:', not via HTML meta tags.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "cloudflare-content-signals-blog",
      "title": "Giving users choice with Cloudflare's new Content Signals Policy",
      "url": "https://blog.cloudflare.com/content-signals-policy/",
      "type": "article",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "Introduces the Content-Signal robots.txt directive. Syntax: 'Content-Signal: search=yes, ai-train=no'. Deployed on the managed robots.txt feature already enabled 'for over 3.8 million domains'. Default omits ai-input because Cloudflare 'doesn't presume user preference'. Includes an EU DSM Art. 4 TDM reservation-of-rights notice in caps. CANDID LIMITS: 'content signals express preferences; they are not technical countermeasures against scraping' and 'Some companies might simply ignore them.' No AI company is named as committing to honor it."
    },
    {
      "id": "cloudflare-crawl-refer-radar",
      "title": "The crawl before the fall… of referrals: understanding AI's impact on content providers",
      "url": "https://blog.cloudflare.com/ai-search-crawl-refer-ratio-on-radar/",
      "type": "dataset",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "Defines crawl-to-refer ratio = requests from a platform's user agents with Content-type: text/html divided by HTML requests whose Referer names that platform. Documents extreme asymmetry (Anthropic ~70,900:1 at time of writing vs near-parity for conventional search). Establishes that AI crawler fetch volume is enormous relative to referral, i.e. crawl budget waste has real cost. Cautions the metric understates disparity because native-app referrals carry no Referer header."
    },
    {
      "id": "cloudflare-crawl-refer-ratio",
      "title": "The crawl before the fall… of referrals: understanding AI's impact on content providers",
      "url": "https://blog.cloudflare.com/ai-search-crawl-refer-ratio-on-radar/",
      "type": "dataset",
      "publisher": "Cloudflare Radar",
      "verified": "2026-08-20",
      "keyFindings": "Methodology: crawler requests answered with Content-type: text/html divided by HTML requests carrying a Referer from the platform's hostnames; UA variants for the same platform aggregated. June 2025 window: Anthropic ~71,000 crawls per referral. Caveat stated in-post: Claude's native app sends no Referer, inflating ratios. Useful as the canonical primary for crawl-to-refer, and as a caution against citing derivative ratio numbers."
    },
    {
      "id": "cloudflare-googlebot-to-gptbot",
      "title": "From Googlebot to GPTBot: who's crawling your site in 2025",
      "url": "https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/",
      "type": "dataset",
      "publisher": "Cloudflare Radar",
      "verified": "2026-08-20",
      "keyFindings": "May 2025 shares of AI-bot traffic: GPTBot 7.7% (+305% requests), ClaudeBot 5.4% (-46%), ChatGPT-User 1.3% (+2,825%), PerplexityBot 0.2%. robots.txt adoption: of 3,816 top-10k domains with a robots.txt, 546 (~14%) had AI-bot directives. GPTBot was both the most-disallowed (312 domains) and the most explicitly allowed (61 domains)."
    },
    {
      "id": "cloudflare-googlebot-to-gptbot-2025",
      "title": "From Googlebot to GPTBot: who's crawling your site in 2025",
      "url": "https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/",
      "type": "article",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "Window May 2024 – May 2025. Combined AI+search crawler traffic grew 18% YoY (peaking 32% above baseline in spring 2025). Googlebot requests +96% YoY, reaching 50% of all crawler traffic (up from 30%). GPTBot +305%, rising from 5% to 30% share among AI crawlers; ChatGPT-User requests up ~2,800%. Bytespider -85% (2nd to 8th place); ClaudeBot -46% in this window. Only 14% of top domains had any AI-crawler-specific robots.txt directive. GPTBot was simultaneously the most-blocked (312 domains) and most-explicitly-allowed (61 domains) crawler."
    },
    {
      "id": "cloudflare-managed-robots-txt-docs",
      "title": "Managed robots.txt — Cloudflare Bots docs",
      "url": "https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/",
      "type": "vendor-doc",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "Current shipped syntax adds a fourth field: 'Content-signal: search=yes, ai-train=no, use=reference'. Definitions: search = 'building a search index and providing search results'; ai-input = 'inputting content into one or more AI models (e.g., retrieval augmented generation, grounding, or other real-time taking of content for generative AI search answers)'; ai-train = 'training or fine-tuning AI models'. Explicit enforcement caveat: 'robots.txt compliance is voluntary' and 'does not prevent crawlers from accessing your content at a technical level. Some crawler operators may disregard your robots.txt directives.' Enforcement requires the separate AI Crawl Control product."
    },
    {
      "id": "cloudflare-markdown-for-agents",
      "title": "Introducing Markdown for Agents",
      "url": "https://blog.cloudflare.com/markdown-for-agents/",
      "type": "vendor-doc",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "Quantifies boilerplate cost at CDN scale. 'This blog post you're reading takes 16,180 tokens in HTML and 3,150 tokens when converted to markdown. That's a 80% reduction in token usage.' 'A simple ## About Us on a page in markdown costs roughly 3 tokens; its HTML equivalent — <h2 class=\"section-title\" id=\"about\">About Us</h2> — burns 12-15'. Names the padding explicitly: 'the <div> wrappers, nav bars, and script tags that pad every real web page and have zero semantic value.' Named consumers already sending Accept: text/markdown: 'Claude Code and OpenCode'. Response carries x-markdown-tokens and x-original-tokens headers."
    },
    {
      "id": "cloudflare-pay-per-crawl",
      "title": "What is pay per crawl?",
      "url": "https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/",
      "type": "vendor-doc",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "Crawlers 'either present payment intent via request headers for successful HTTP 200 access, or receive an HTTP 402 Payment Required response with pricing.' Minimum price $0.001/crawl. WAF/Bot Management block rules override pay-per-crawl's charge behaviour — i.e. an existing WAF rule silently converts a monetizable crawl into a hard block."
    },
    {
      "id": "cloudflare-perplexity-stealth-crawlers",
      "title": "Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives",
      "url": "https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/",
      "type": "article",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "Published 2025-08-04. Controlled experiment: brand-new unpublished, unindexed domains with robots.txt universal disallow; Perplexity still returned detailed page content. Perplexity used a generic Chrome-on-macOS impersonating user agent rotating undisclosed IPs/ASNs. Volumes: declared PerplexityBot 20–25M requests/day; stealth crawler 3–6M requests/day across tens of thousands of domains. Control: OpenAI's ChatGPT fetched robots.txt and stopped when disallowed. Perplexity publicly disputed the findings."
    },
    {
      "id": "cloudflare-radar-2025-year-review",
      "title": "Cloudflare Radar 2025 Year in Review",
      "url": "https://blog.cloudflare.com/radar-2025-year-in-review/",
      "type": "dataset",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "Data through 2025-12-02. AI bots accounted for 4.2% of HTML requests across Cloudflare's network in 2025 (low 2.4% early April, high 6.4% late June). Googlebot alone was 4.5% of HTML requests — slightly more than all other AI bots combined. GPTBot ~7.5% of verified bot traffic, Bingbot ~6%. ChatGPT-User grew ~16x at peak; PerplexityBot ~3.5x above year-start; ClaudeBot ~10% above year-start by December. AI crawlers were the most frequently FULLY disallowed user agents in robots.txt (GPTBot, ClaudeBot, CCBot highest), while Googlebot/Bingbot directives leaned to partial disallow. Top generative AI services by popularity: ChatGPT, Perplexity, Claude, GitHub Copilot, Gemini, Windsurf, Grok, DeepSeek."
    },
    {
      "id": "cloudflare-radar-ai-insights",
      "title": "Cloudflare Radar — AI Insights",
      "url": "https://radar.cloudflare.com/ai-insights",
      "type": "dataset",
      "publisher": "Cloudflare Radar",
      "verified": "2026-08-20",
      "keyFindings": "Live per-user-agent AI crawler traffic-share and robots.txt dashboard. Returned HTTP 403 to automated fetch (bot protection), so we could not extract figures directly; the underlying data is reported in cloudflare-ai-crawler-purpose. Listed for human verification only."
    },
    {
      "id": "cloudflare-web-bot-auth",
      "title": "Web Bot Auth — Cloudflare bot verification",
      "url": "https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/",
      "type": "vendor-doc",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "Production implementation of cryptographic bot identity: Ed25519-signed requests per RFC 9421 HTTP Message Signatures, with Signature-Input, Signature and Signature-Agent headers, and a public key directory hosted at /.well-known/http-message-signatures-directory (JWKS). This is the emerging replacement for UA/IP allow-listing when deciding whether to let an AI agent through a WAF."
    },
    {
      "id": "commoncrawl-ccbot",
      "title": "CCBot — Common Crawl",
      "url": "https://commoncrawl.org/ccbot",
      "type": "vendor-doc",
      "publisher": "Common Crawl Foundation",
      "verified": "2026-08-20",
      "keyFindings": "UA: 'CCBot/2.0 (https://commoncrawl.org/faq/)'. Canonical block snippet published by the operator: 'User-agent: CCBot' / 'Disallow: /'. Warns of spoofed CCBot traffic and recommends reverse-DNS verification against published IP ranges. Page does not state crawl-delay support, and does not itself frame the corpus as AI training data (though C4/RefinedWeb/Dolma derive from it — see consent-in-crisis-arxiv)."
    },
    {
      "id": "conductor",
      "title": "Conductor platform",
      "url": "https://www.conductor.com/",
      "type": "vendor-doc",
      "publisher": "Conductor",
      "verified": "2026-08-20",
      "keyFindings": "Conductor Intelligence (visibility across ChatGPT/Gemini/Copilot/Claude), Conductor Creator, Conductor Monitoring ('24/7 monitoring tracks how AI bots crawl your site' — log/edge telemetry plus alerts), Conductor AgentStack. The only technical surface is passive bot-crawl monitoring; no documented llms.txt/schema/agent-protocol conformance audits."
    },
    {
      "id": "consent-in-crisis-arxiv",
      "title": "Consent in Crisis: The Rapid Decline of the AI Data Commons",
      "url": "https://arxiv.org/abs/2407.14933",
      "type": "study",
      "publisher": "Data Provenance Initiative / MIT (Longpre et al., NeurIPS 2024)",
      "verified": "2026-08-20",
      "keyFindings": "First large-scale longitudinal audit of robots.txt + ToS consent across 14,000 web domains. 'in a single year (2023-2024) there has been a rapid crescendo of data restrictions from web sources, rendering ~5%+ of all tokens in C4, or 28%+ of the most actively maintained, critical sources in C4, fully restricted from use. For Terms of Service crawling restrictions, a full 45% of C4 is now restricted.' Also documents 'acute differences in restrictions on AI developers' (OpenAI far more blocked than others) and 'general inconsistencies between websites' expressed intentions in their Terms of Service and their robots.txt'. Diagnoses robots.txt as 'ineffective web protocols, not designed to cope with the widespread re-purposing of the internet for AI'."
    },
    {
      "id": "contentsignals",
      "title": "contentsignals.org",
      "url": "https://contentsignals.org/",
      "type": "spec",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "Canonical policy home. Fetched successfully but served only a bare 'Content Signals' heading to the markdown extractor (JS-rendered); policy text was not retrievable, so treated as unverified and the Cloudflare blog post is used as the citable source."
    },
    {
      "id": "contentsignals-org",
      "title": "Content Signals Policy (canonical policy site)",
      "url": "https://contentsignals.org/",
      "type": "draft-spec",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "URL resolves but the body is client-side rendered; automated fetch returned only the 'Content Signals' heading with no policy text. Cited as the policy's canonical home, but all substantive text in this dossier comes from cloudflare-content-signals-blog and cloudflare-managed-robots-txt-docs instead."
    },
    {
      "id": "cseo-bench-arxiv",
      "title": "C-SEO Bench: Does Conversational SEO Work?",
      "url": "https://arxiv.org/abs/2506.11097",
      "type": "study",
      "publisher": "Puerto, Gubri, Green, Oh, Yun (arXiv; NeurIPS 2025 Datasets & Benchmarks)",
      "verified": "2026-08-20",
      "keyFindings": "Replication failure for GEO-style content edits: 9 C-SEO methods across 6 domains and 1,921 queries. 'Most current C-SEO methods are not only largely ineffective but also frequently have a negative impact on document ranking, which is opposite to what is expected.' Traditional retrieval-ranking SEO (moving the source to context position 1) was ~7.6x more effective in the retail domain than the best C-SEO method. Effectiveness further degrades as adoption rises ('congested and zero-sum nature'). Per the critical survey's reading, 'only three of 54 method–domain combinations are significantly positive in the main experiment; none is positive in question answering.'"
    },
    {
      "id": "data-action-adoption-search",
      "title": "GitHub code search: \"data-action-type\" \"data-action-label\"",
      "url": "https://github.com/search?q=%22data-action-type%22+%22data-action-label%22&type=code",
      "publisher": "GitHub",
      "type": "repo",
      "keyFindings": "(GitHub code-search query link — interactive only, result counts recorded at research time.) 40 total results; the large majority are files within a single repository (magnifito/website — the AIO framework's own site: HomePage.astro, ContactPage.astro, ServicesPage.astro, LeadMagnet.astro, ConversionBoosters.astro and others). No framework, vendor, or standards body among the results. Adoption is self-referential, not a community convention.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "debugbear-llms-txt-audit-doc",
      "title": "Lighthouse: llms.txt does not follow recommendations",
      "url": "https://www.debugbear.com/docs/agentic-browsing/llms-txt-does-not-follow-recommendations",
      "type": "vendor-doc",
      "publisher": "DebugBear",
      "verified": "2026-08-20",
      "keyFindings": "Independent restatement of the Lighthouse audit (v13.3.0+), matching the source code I read: requires H1 at top, at least one [text](URL) markdown link, and >50 characters. Quotes the spec's 'An H1 with the name of the project or site. This is the only required section.' Notes llms.txt 'is not currently widely adopted by AI services.'"
    },
    {
      "id": "deque-axe-rule-page-image-alt",
      "title": "axe rule help page: image-alt (Deque University)",
      "url": "https://dequeuniversity.com/rules/axe/4.10/image-alt",
      "type": "vendor-doc",
      "publisher": "Deque Systems",
      "verified": "2026-08-20",
      "keyFindings": "Per-rule dossier template: rule ID + ruleset version + user impact; compliance mapping (WCAG 2.0/2.1/2.2 level, Section 508, Trusted Tester, EN 301 549); disabilities affected; normative success-criteria text with MUST language; 'How to fix'; 'Why it matters'; algorithm explanation; and a Resources block linking W3C techniques (H37, F65). Every rule page cites the normative standard it derives from — the closest existing analogue to a per-audit evidence dossier."
    },
    {
      "id": "distracted-irrelevant",
      "title": "Large Language Models Can Be Easily Distracted by Irrelevant Context",
      "url": "https://arxiv.org/abs/2302.00093",
      "type": "study",
      "publisher": "Shi et al., ICML 2023 (arXiv 2302.00093)",
      "verified": "2026-08-20",
      "keyFindings": "Introduces GSM-IC; finds \"the model performance is dramatically decreased when irrelevant information is included\" in the prompt, mitigated only partially by self-consistency and explicit ignore-instructions. Grounds the claim that boilerplate/duplicate/hidden text in an ingested page degrades answer quality, not just cost."
    },
    {
      "id": "dom-downsampling-paper",
      "title": "Beyond Pixels: Exploring DOM Downsampling for LLM-Based Web Agents",
      "url": "https://arxiv.org/html/2508.04412v1",
      "type": "study",
      "publisher": "arXiv (Thassilo M. Schiepanski, 2025)",
      "verified": "2026-08-20",
      "keyFindings": "Quantifies DOM bloat as an agent-context problem. 'Some real world DOMs surpass the size of a megabyte' ≈ 1e6 input tokens. The proposed D2Snap downsampling reaches ~1e3–1e4 tokens via hierarchical merging of containers, Markdown conversion of content elements, and semantic-aware attribute filtering that selectively preserves alt, href and aria-*. Ablation result that matters most here: DOM hierarchy 'emerged as the strongest among those features' for LLM performance. Best configs hit 73% task success vs 65% for the grounded-GUI (screenshot) baseline."
    },
    {
      "id": "dries-buytaert-markdown-llmstxt",
      "title": "Markdown, llms.txt and AI crawlers",
      "url": "https://dri.es/markdown-llms-txt-and-ai-crawlers",
      "type": "study",
      "publisher": "Dries Buytaert (Drupal founder)",
      "verified": "2026-08-20",
      "keyFindings": "One month of Cloudflare logs, January 2026. llms.txt: 'My site received 52 requests for it last month. Every one came from SEO audit tools. Not a single request came from an AI answer engine or crawler.' MARKDOWN: GPTBot requested markdown 34.8% of the time, ChatGPT-User 0.1% — but discovery was via dedicated .md URLs only: 'No AI crawler uses content negotiation. Not one.' Serving markdown did not reduce bot load; crawl volume rose ~7%. Also: ~1,241 pages fetched per citation returned. NOTE the direct conflict with Evil Martians on content negotiation, which is best explained by date (Jan vs May–Jul 2026) and by crawlers-vs-coding-agents population."
    },
    {
      "id": "duckduckgo-duckassistbot",
      "title": "DuckAssistBot — DuckDuckGo Help Pages",
      "url": "https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot/",
      "type": "vendor-doc",
      "publisher": "DuckDuckGo",
      "verified": "2026-08-20",
      "keyFindings": "UA token: 'DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)'. 'DuckAssistBot is a web crawler for DuckDuckGo Search that crawls pages in real-time for our AI-assisted answers' and 'This data is not used in any way to train AI models.' Honors robots.txt; disallow 'will take effect after 72 hours'. 'Opting out of DuckAssistBot does not impact organic search rankings.'"
    },
    {
      "id": "ec-tdm-protocols-consultation",
      "title": "Commission launches consultation on protocols for reserving rights from text and data mining under the AI Act and the GPAI Code of Practice",
      "url": "https://digital-strategy.ec.europa.eu/en/consultations/commission-launches-consultation-protocols-reserving-rights-text-and-data-mining-under-ai-act-and",
      "type": "article",
      "publisher": "European Commission (DG CONNECT)",
      "verified": "2026-08-20",
      "keyFindings": "Consultation opened 1 December 2025, closed 23 January 2026 (extended), info session 9 December 2025. Names only robots.txt 'and subsequent IETF versions of this standard' as committed protocols. The Commission will 'publish the list of generally-agreed machine-readable opt-out solutions' and review it at least every two years. As of this research date no such list naming TDM-Rep has been published — TDM-Rep's legal weight is prospective, not established."
    },
    {
      "id": "eia-iclr25",
      "title": "EIA: Environmental Injection Attack on Generalist Web Agents",
      "url": "https://arxiv.org/abs/2409.11295",
      "type": "study",
      "publisher": "arXiv / ICLR 2025",
      "verified": "2026-08-20",
      "keyFindings": "Injects content into the page environment that blends into the surrounding site. Up to 70% ASR for stealing specific PII, 16% for extracting the full user request, over 177 Mind2Web action steps. Authors report EIA is hard to detect and that well-adapted injections survive human inspection — i.e. detection has to be mechanical, not eyeballed."
    },
    {
      "id": "eu-ai-act-article-53",
      "title": "EU AI Act — Article 53: Obligations for providers of general-purpose AI models",
      "url": "https://artificialintelligenceact.eu/article/53/",
      "type": "spec",
      "publisher": "EU AI Act (artificialintelligenceact.eu consolidated text)",
      "verified": "2026-08-20",
      "keyFindings": "Art. 53(1)(c) requires GPAI providers to 'put in place a policy to comply with Union law on copyright and related rights, and in particular to identify and comply with, including through state-of-the-art technologies, a reservation of rights expressed pursuant to Article 4(3) of Directive (EU) 2019/790'. The Article itself does NOT name any specific machine-readable protocol."
    },
    {
      "id": "eu-gpai-code-of-practice",
      "title": "EU AI Act: General-Purpose AI Code of Practice — Final version (10 July 2025)",
      "url": "https://code-of-practice.ai/",
      "type": "spec",
      "publisher": "European Commission / GPAI Code of Practice drafting process",
      "verified": "2026-08-20",
      "keyFindings": "Copyright Chapter Measure 1.3 ('Identify and comply with rights reservations when crawling the World Wide Web') commits signatories to 'employ web-crawlers that read and follow instructions expressed in accordance with the Robot Exclusion Protocol (robots.txt), as specified in the Internet Engineering Task Force (IETF) Request for Comments No. 9309', plus 'other appropriate machine-readable protocols... state-of-the-art, including technically implementable, and widely adopted by rightsholders' determined through an EU-facilitated process. robots.txt/RFC 9309 is the ONLY protocol named."
    },
    {
      "id": "evil-martians-llm-traffic",
      "title": "Which AI actually reads your site? Two months of LLM traffic, measured",
      "url": "https://evilmartians.com/chronicles/which-ai-actually-reads-your-site-two-months-of-llm-traffic-measured",
      "type": "study",
      "publisher": "Evil Martians",
      "verified": "2026-08-20",
      "keyFindings": "Best-instrumented study found. evilmartians.com, early May – early July 2026, ~268,000 agent requests vs ~107,000 human pageviews, measured at the edge (Netlify edge functions reading raw User-Agent and Accept headers pre-JS). llms.txt: ~770 direct fetches but only 37 from named AI assistants; the rest 'a zoo' of search crawlers and 'llms.txt-curious fetchers'; of 117 pageviews referred from /llms.txt, >90% came from one client sending a frozen Chrome/111.0 UA (bot signature). MARKDOWN: agents took markdown on ~15% of reads (~40,000 fetches), wildly uneven by agent — Claude Code 76%, ChatGPT-User 0.1%. Concludes content negotiation 'is the mechanism actually delivering Markdown to the client that wants it (Claude Code), it requires zero site-specific knowledge from the agent.'"
    },
    {
      "id": "fastmcp-openapi",
      "title": "FastMCP — OpenAPI integration",
      "url": "https://gofastmcp.com/integrations/openapi",
      "type": "vendor-doc",
      "publisher": "FastMCP",
      "verified": "2026-08-21",
      "keyFindings": "\"Creates MCP components using a variety of metadata from the OpenAPI spec, such as incorporating the OpenAPI description into the MCP component description\", and derives the tool name from `operationId`."
    },
    {
      "id": "fern-api-catalog",
      "title": "Fern — api-catalog",
      "url": "https://buildwithfern.com/learn/docs/ai-features/api-catalog",
      "type": "vendor-doc",
      "publisher": "Fern",
      "keyFindings": "Documentation platform that publishes `/.well-known/api-catalog` for the sites it hosts.",
      "verified": "2026-08-24"
    },
    {
      "id": "firecrawl-docs",
      "title": "Firecrawl documentation",
      "url": "https://docs.firecrawl.dev",
      "type": "vendor-doc",
      "publisher": "Firecrawl",
      "keyFindings": "HTML-to-markdown extraction service; one of the converters whose table round-trip this audit measures.",
      "verified": "2026-08-20"
    },
    {
      "id": "firecrawl-scrape-api",
      "title": "Scrape endpoint API reference",
      "url": "https://docs.firecrawl.dev/api-reference/endpoint/scrape",
      "type": "vendor-doc",
      "publisher": "Firecrawl",
      "verified": "2026-08-20",
      "keyFindings": "onlyMainContent default: true (excludes headers, navs, footers via HTML-level filtering, no LLM); removeBase64Images default: true (\"Removes all base 64 images from the markdown output\"); blockAds default: true; the cleaned html format \"Removes <script>, <style>, <noscript>, <meta>, and <head> tags\". A commercial extraction vendor defaults to deleting base64 images and chrome — direct evidence these are treated as pure token waste."
    },
    {
      "id": "firecrawl-scrape-feature",
      "title": "Scrape — output formats",
      "url": "https://docs.firecrawl.dev/features/scrape",
      "type": "vendor-doc",
      "publisher": "Firecrawl",
      "verified": "2026-08-20",
      "keyFindings": "Offers markdown, html (\"cleaned version of the page's HTML\"), rawHtml (\"unmodified HTML as received\"), summary, links, json. Confirms the three-way distinction (raw HTML / cleaned HTML / markdown) that a markup-to-content token ratio measures."
    },
    {
      "id": "format-as-prior-arxiv",
      "title": "Format as a Prior: Quantifying and Analyzing Bias in LLMs for Heterogeneous Data",
      "url": "https://arxiv.org/html/2508.15793v1",
      "type": "study",
      "publisher": "Liu, Xu, Pi, Li, Zhong, Zhu, Liu, Qian (arXiv, Aug 2025 / rev. Jan 2026)",
      "verified": "2026-08-20",
      "keyFindings": "Direct counter-evidence for the comparison-table signal: 'our cross-model analysis reveals a clear preference hierarchy: semantically rich formats such as texts and KGs are consistently favored over visually structured ones like infoboxes and tables.' Reported format-preference rates vs plain text: tables 0.398, knowledge graphs 0.336, infoboxes 0.235 — i.e. when the same fact is available as prose or as a table, models preferentially draw on the prose. Format bias is 'consistent across model families, driven by information richness, structure quality, and representation type.'"
    },
    {
      "id": "freecodecamp-webmcp-zero-adoption",
      "title": "A Developer's Guide to WebMCP: Shipping a 0% Adoption Standard",
      "url": "https://www.freecodecamp.org/news/a-developers-guide-to-webmcp/",
      "type": "study",
      "publisher": "freeCodeCamp",
      "verified": "2026-08-20",
      "keyFindings": "Crawled 111,076 top websites (measurement May 2026) and found 'exactly zero' domains with WebMCP in production — 0% of the top 200k. Author shipped WebMCP tools on two live sites: 'In the 93 days since [shipping], zero external AI agents have called searchPosts, listPosts, or getAuthorContext. Zero.' Also shipped a `.well-known/webmcp` JSON manifest on citability.dev declaring four tools — 'five days later, still zero agent calls.' Vercel logs showed only 'routine crawler traffic and 404 bot probes'."
    },
    {
      "id": "gemini-computer-use-docs",
      "title": "Computer use — Gemini API",
      "url": "https://ai.google.dev/gemini-api/docs/computer-use",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-20",
      "keyFindings": "COUNTER-EVIDENCE. 'Using screenshots, the model can \"see\" a computer screen, and \"act\" by generating specific UI actions like mouse clicks and keyboard inputs.' Inputs are user request + screenshot + recent action history. No DOM or accessibility-tree input."
    },
    {
      "id": "gemini-for-home",
      "title": "Gemini for Home",
      "url": "https://blog.google/products/google-nest/gemini-for-home/",
      "publisher": "Google (The Keyword)",
      "type": "announcement",
      "keyFindings": "Dated 2025-08-20. 'Over time, Gemini for Home will replace Google Assistant on existing speakers and displays, with free and paid versions.' The named consumer of speakable is on a retirement path.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "gemini-function-calling",
      "title": "Gemini API — function calling",
      "url": "https://ai.google.dev/gemini-api/docs/function-calling",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-21",
      "keyFindings": "Consumes only \"a subset of the OpenAPI schema\" and instructs \"Use descriptive names without spaces or special characters\" for function names."
    },
    {
      "id": "geo-16-study",
      "title": "GEO-16: a framework for generative-engine citation",
      "url": "https://arxiv.org/abs/2509.10762",
      "type": "study",
      "publisher": "arXiv",
      "keyFindings": "Audited 1,702 citations across Brave Summary, Google AI Overviews and Perplexity: \"pillars related to Metadata and Freshness, Semantic HTML, and Structured Data showed the strongest associations with citation\". Pillar-level, not schema-type-level.",
      "verified": "2026-08-21"
    },
    {
      "id": "geo-at-scale",
      "title": "Generative Engine Optimization at Scale: Measuring Brand Visibility Across AI Search Engines",
      "url": "https://arxiv.org/abs/2606.20065",
      "publisher": "arXiv",
      "type": "study",
      "verified": "2026-08-20",
      "keyFindings": "About 78% of citations go to corporate websites; ranked 'best-of' listicles are the most-cited format at ~21% of citations; visibility tracks brand stature (73% for household names, 44% mid-market, 11% niche). No on-site trust-badge or credential signal is identified as a correlate.",
      "accessed": "2026-08-21"
    },
    {
      "id": "geo-critical-survey-arxiv",
      "title": "Optimizing Visibility in Generative Engines: A Critical Survey of Generative Engine Optimization (2023–2026)",
      "url": "https://arxiv.org/html/2607.14035v1",
      "type": "study",
      "publisher": "arXiv",
      "verified": "2026-08-21",
      "keyFindings": "Systematic critique of the GEO literature. On the original paper: the 'up to 40%' gain applies only when 'a source is already present in a five-document context', effects are 'conditional on retrieval' and do not establish durable traffic effects; results are 'valid within its experimental setting' but do not generalize to production systems. Replication status by tactic — strong: topical relevance and position ('the most reproducible levers'), extractable evidence (figures, definitions, quotations, references) with moderate-to-strong support; moderate/mixed: dates and recency (help time-sensitive queries, 'lack universality'), statistics (gains observed but 'factuality not guaranteed'), HTML structure (heterogeneous); poor: generic heuristics ('only three of 54 method–domain combinations significantly positive'), keyword stuffing ('null or negative'), formatting alone. Methodological criticisms: LLM-as-judge introduces 'model dependence, stylistic bias, and circularity', worst 'when the same model generates the rewrite, the response, and the score'. SAGEO Arena found body-only optimization REDUCES average top-20 presence by ~9%, contradicting fixed-context gains. Cross-engine citation overlap is low (26% Bing–Perplexity; Jaccard <0.2). Critically for this domain: authority signals (authorship, E-E-A-T credentials, brand) are 'not systematically studied'; authority/credibility effects are 'weak and unstable' and 'may conflict with credibility'; brand signals 'not independently evaluated'."
    },
    {
      "id": "geo-dominate-ai-search-arxiv",
      "title": "Generative Engine Optimization: How to Dominate AI Search",
      "url": "https://arxiv.org/abs/2509.08919",
      "type": "study",
      "publisher": "Chen, Wang, Chen, Koudas (arXiv, September 2025)",
      "verified": "2026-08-20",
      "keyFindings": "Large-scale controlled comparison of AI Search vs Google across verticals, languages and query paraphrases. 'Our key findings reveal that AI Search exhibit a systematic and overwhelming bias towards Earned media (third-party, authoritative sources) over Brand-owned and Social content.' Engines 'differ significantly from each other in their domain diversity, freshness, cross-language stability, and sensitivity to phrasing.' Its first strategic recommendation is to 'engineer content for machine scannability and justification' — but the paper's own measured findings emphasize earned media and authority over on-page structure, i.e. structure is a secondary lever."
    },
    {
      "id": "geo-kdd24",
      "title": "GEO: Generative Engine Optimization",
      "url": "https://arxiv.org/abs/2311.09735",
      "publisher": "arXiv / KDD 2024",
      "type": "study",
      "verified": "2026-08-21",
      "keyFindings": "Boosts visibility up to 40%; nine methods tested (Quotation Addition +41%, Statistics +33%, Fluency +29%, Cite Sources +28%, Technical Terms +19%, Authoritative +17%, Easy-to-Understand +14%, Unique Words +6%, Keyword Stuffing -9%). Explicitly scoped to on-page text, not metadata or site-level authority; does not study about pages, credentials, or trust badges.",
      "accessed": "2026-08-21"
    },
    {
      "id": "geo-kdd24-trust",
      "title": "GEO: Generative Engine Optimization",
      "url": "https://arxiv.org/abs/2311.09735",
      "publisher": "arXiv / KDD 2024",
      "type": "study",
      "verified": "2026-08-21",
      "keyFindings": "'Authoritative' rewriting measured at +17% visibility; Quotation Addition +41%, Statistics +33%, Cite Sources +28%, Keyword Stuffing -9%. Scope limited to textual content edits — trust badges, testimonials, awards and logo grids are not tested.",
      "accessed": "2026-08-21"
    },
    {
      "id": "geo-lint-repo",
      "title": "geo-lint",
      "url": "https://github.com/IJONIS/geo-lint",
      "type": "article",
      "publisher": "IJONIS — 39 stars, MIT, pushed 2026-03-19",
      "verified": "2026-08-20",
      "keyFindings": "'The first open-source linter for GEO'; ~97 content rules emitting structured violations each with a machine-readable suggestion field, designed for an autonomous agent lint-fix loop, shipped as a Claude Code skill. Rule-engine ergonomics are strong (this is the design we should match for fix-loops), but rules are content-style heuristics with no per-rule evidence, no grades, and no named consumers. Not updated since March 2026."
    },
    {
      "id": "geo-optimizer-skill-repo",
      "title": "GEO Optimizer / GeoReady (geo-optimizer-skill)",
      "url": "https://github.com/Auriti-Labs/geo-optimizer-skill",
      "type": "article",
      "publisher": "Auriti Labs — 723 stars, MIT, pushed 2026-08-17",
      "verified": "2026-08-20",
      "keyFindings": "The largest OSS competitor by adoption. Python CLI + library + MCP server + Astro integration; '16 CLI commands, 8 scoring categories, 47 research-backed methods, 7 output formats, 1,720 tests', scoring a site 0-100. Cites an academic foundation inline (arXiv:2311.09735 KDD 2024, arXiv:2510.11438 labelled ICLR 2026). Gaps: the '47 research-backed methods' are cited collectively at README level, not per check; no per-audit evidence grade, no consumer list, no counter-evidence section, no published weight table; it is open-core with a hosted product (geoready.dev pricing/signup), and it cites a vendor blog (llmrefs.com) for its headline '900M weekly users' figure."
    },
    {
      "id": "geo-paper",
      "title": "GEO: Generative Engine Optimization",
      "url": "https://arxiv.org/html/2311.09735v3",
      "publisher": "arXiv (Aggarwal et al., KDD 2024)",
      "type": "study",
      "keyFindings": "Nine tested optimization methods, all content-level: quotation addition (+41%), statistics addition (+39%), cite sources (+28%), fluency optimization, easy-to-understand, authoritative, unique words, technical terms, keyword stuffing. No HTTP header, security header, security.txt, page-speed or resource-hint lever was tested or discussed.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "geo-paper-arxiv",
      "title": "GEO: Generative Engine Optimization (Aggarwal, Murahari, Rajpurohit, Kalyan, Narasimhan, Deshpande)",
      "url": "https://arxiv.org/abs/2311.09735",
      "type": "study",
      "publisher": "arXiv (Princeton University, IIT Delhi, Georgia Tech, Allen Institute for AI)",
      "verified": "2026-08-21",
      "keyFindings": "Read the full PDF directly. GEO-BENCH = 10K queries (8K/1K/1K splits), 9 sources, top-5 Google results per query, response generated by gpt-3.5-turbo in a BingChat-mimicking 2-step pipeline; subjective metrics scored by G-Eval (LLM-as-judge). Table 1 (baseline 19.3 on both metrics): Quotation Addition 27.2 PAWC (+40.9%) / 24.7 SI (+28.0%); Statistics Addition 25.2 (+30.6%) / 23.7 (+22.8%); Cite Sources 24.6 (+27.5%) / 21.9 (+13.5%); Fluency Opt 24.7 / 21.9; Technical Terms 22.7 / 21.4; Authoritative 21.3 (+10.4%) / 22.9; Easy-to-Understand 22.0 / 20.5; Unique Words 20.5 / 20.4; Keyword Stuffing 17.7 (-8.3%) / 20.2. Paper states of Authoritative tone: 'to the contrary we find no significant improvement, demonstrating that Generative Engines are already somewhat robust to such changes.' Table 3 (all sources optimized simultaneously) shows gains are ZERO-SUM by SERP rank: Cite Sources rank-1 -30.3%, rank-5 +115.1%; Quotation rank-1 -22.9%, rank-5 +99.7%; Statistics rank-1 -20.6%, rank-5 +97.9%. Table 5 (real Perplexity.ai deployment, 200 samples, baseline 24.1 PAWC / 24.7 SI): Quotation 29.1 (+20.7%) / 32.1 (+30.0%); Statistics 26.2 (+8.7%) / 33.9 (+37.2%); Cite Sources 26.8 (+11.2%) / 19.0 (MINUS 23.1% — the only high-performer to go negative on a live engine). Section 2.2.2: modifications were produced by prompting GPT-3.5, i.e. quotes/statistics/citations were LLM-generated, not verified real."
    },
    {
      "id": "geo-paper-kdd",
      "title": "GEO: Generative Engine Optimization — Proceedings of the 30th ACM SIGKDD Conference (KDD '24)",
      "url": "https://dl.acm.org/doi/10.1145/3637528.3671900",
      "type": "study",
      "publisher": "ACM",
      "verified": "2026-08-20",
      "keyFindings": "Peer-reviewed venue of record for the GEO paper (KDD 2024, Barcelona). Listed for citation completeness; content verified via the arXiv version instead (ACM DL page not fetched)."
    },
    {
      "id": "geo-sfe-structural-arxiv",
      "title": "Structural Feature Engineering for Generative Engine Optimization: How Content Structure Shapes Citation Behavior",
      "url": "https://arxiv.org/html/2603.29979v1",
      "type": "study",
      "publisher": "Yu, Yang, Ding, Sato (arXiv, March 2026)",
      "verified": "2026-08-21",
      "keyFindings": "The only controlled study isolating STRUCTURE from semantics. 200 articles across 6 domains, 377 queries, 2,400 test cases, 6 engines (Google SGE, Bing Chat, Perplexity, Phind, ChatGPT, Claude), semantic preservation verified (mean Bge-m3 similarity 0.843). Overall citation improvement 17.3% (p<0.001); Visibility Score 0.411→0.483; +18.5 average subjective G-Eval. Decomposition: macro-structure (document architecture/headings, target heading depth 3–5 levels, internal linking 0.15–0.20 links per concept) = 44.9% of gains, up to +19.2% citation rate on search-then-synthesize engines, ablation −3.5pp; meso-structure (information chunking; paragraph length 150–300 words; 25–35% structured elements such as lists and tables) = 39.7% of gains, +14.0% on iterative-refinement engines, ablation −3.1pp; micro-structure (emphasis markers at 5–10% of content, position-weighted keyword placement 2.0x initial / 1.5x boundary) = 15.4% of gains, ablation −1.2pp. Caveat: the paper reports no null or negative per-feature results, and does not isolate question-form headings, FAQ blocks, definition sentences or dates individually."
    },
    {
      "id": "geo-survey",
      "title": "Optimizing Visibility in Generative Engines: A Critical Survey of Generative Engine Optimization (2023-2026)",
      "url": "https://arxiv.org/abs/2607.14035",
      "publisher": "arXiv",
      "type": "study",
      "verified": "2026-08-20",
      "keyFindings": "Finds 'topical relevance and context position are the most reproducible levers'; warns 'generic heuristics transfer poorly', 'competition can erode individual gains', and 'citation-oriented rewrites can impair retrieval'. GEO gains are conditional on a source already being in context and 'establish neither organic discoverability nor durable traffic effects'.",
      "accessed": "2026-08-21"
    },
    {
      "id": "geo16-study",
      "title": "AI Answer Engine Citation Behavior: An Empirical Analysis of the GEO16 Framework",
      "url": "http://arxiv.org/abs/2509.10762",
      "publisher": "arXiv",
      "type": "study",
      "keyFindings": "1,702 citations across Brave Summary, Google AI Overviews and Perplexity; 1,100 URLs audited. 'Pillars related to Metadata and Freshness, Semantic HTML, and Structured Data showed the strongest associations with citation.' Pillar-level only — does not isolate HowTo or any specific type.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "gfm-tables-extension",
      "title": "GitHub Flavored Markdown Spec — tables extension",
      "url": "https://github.github.com/gfm/#tables-extension-",
      "type": "spec",
      "publisher": "GitHub",
      "keyFindings": "The markdown table grammar every HTML-to-markdown converter targets: one header row, one delimiter row, no rowspan, no colspan, no nested block content in a cell.",
      "verified": "2026-08-20"
    },
    {
      "id": "github-adoption-x-ai-instructions",
      "title": "GitHub code search: x-ai-instructions vs x-openai-isConsequential",
      "url": "https://github.com/search?q=%22x-ai-instructions%22&type=code",
      "publisher": "GitHub (code search API)",
      "type": "study",
      "keyFindings": "Measured via GitHub REST search/code API. 'x-ai-instructions': total_count=54. Control field 'x-openai-isConsequential' (genuinely documented by OpenAI): total_count=6,464 — a ~120x gap. Grouping the top 40 x-ai-instructions matches by repo: 31 in api-evangelist/done (a blogger's notes archive), 2 in api-evangelist/providers, and 1 in magnifito/website (this framework's own site); the rest are single-file mentions in personal roadmaps and handbooks. No vendor, spec, or toolchain repo appears.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "github-docs-link-index",
      "title": "GitHub Docs homepage HTML (live head inspection)",
      "url": "https://docs.github.com/en",
      "publisher": "GitHub",
      "type": "vendor-doc",
      "keyFindings": "Only real-world llms link tag found across five major publishers: <link rel=\"index\" type=\"text/markdown\" href=\"https://docs.github.com/llms.txt\" title=\"LLM-friendly index of all GitHub Docs content\">. Different rel, type and target than the audit requires; docs.github.com/llms-full.txt returns 404.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "gmail-one-click-action",
      "title": "One Click Action reference (Gmail markup)",
      "url": "https://developers.google.com/workspace/gmail/markup/reference/one-click-action",
      "publisher": "Google Workspace",
      "type": "vendor-doc",
      "keyFindings": "'One Click actions currently supported in Gmail are: ConfirmAction [and] SaveAction.' Live, no deprecation notice. This is email-body markup meaning 'render a confirm button', not web-page markup asserting a completed transaction — adjacent to, not supportive of, the audit's claim.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "google-ai-features",
      "title": "AI Features and Your Website",
      "url": "https://developers.google.com/search/docs/appearance/ai-features",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-21",
      "keyFindings": "Primary counter-evidence for most GEO checklist signals. Google states verbatim: 'There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary.' And: 'You don't need to create new machine readable files, AI text files, or markup to appear in these features. There's also no special schema.org structured data that you need to add.' Guidance reduces to core Search fundamentals — meet technical requirements, follow policies, create helpful reliable people-first content, ensure textual content is available. The document does not mention E-E-A-T in connection with AI features."
    },
    {
      "id": "google-ai-features-2",
      "title": "AI features and your website",
      "url": "https://developers.google.com/search/docs/appearance/ai-features",
      "publisher": "Google Search Central",
      "type": "vendor-doc",
      "keyFindings": "Complete list of controls over Google AI features is nosnippet/data-nosnippet/max-snippet/noindex. No llms.txt or llms-full.txt anywhere.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "google-ai-features-controls",
      "title": "Google Search's AI features and your website",
      "url": "https://developers.google.com/search/docs/appearance/ai-features",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-21",
      "keyFindings": "COUNTER-EVIDENCE to a common misconception: Google-Extended does NOT control AI Overviews or AI Mode. 'robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search.' Publisher controls for AI features are nosnippet, data-nosnippet, max-snippet and noindex — not Google-Extended, which governs only Gemini-app training and Vertex grounding."
    },
    {
      "id": "google-ai-features-doc",
      "title": "AI Features and Your Website",
      "url": "https://developers.google.com/search/docs/appearance/ai-features",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-21",
      "keyFindings": "Explicit: \"You don't need to create new machine readable files, AI text files, or markup to appear in these features\" and \"There's also no special schema.org structured data that you need to add.\" Advises structured data should match visible text. Primary counter-evidence against any AI-citation-uplift claim for schema."
    },
    {
      "id": "google-ai-features-docs",
      "title": "AI features and your website — Google Search Central",
      "url": "https://developers.google.com/search/docs/appearance/ai-features",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-21",
      "keyFindings": "BROAD COUNTER-EVIDENCE. 'There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary.' The only structural advice is 'making sure that important content is available in textual form'. No mention of semantic HTML, heading hierarchy, landmarks, or accessibility as AI-feature requirements. Any audit claiming a documented AI-search ranking effect for semantic markup is overclaiming."
    },
    {
      "id": "google-ai-features-guidance",
      "title": "AI features and your website (Google Search Central)",
      "url": "https://developers.google.com/search/docs/appearance/ai-features",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-21",
      "keyFindings": "Category-level counter-evidence from the largest consumer. Google states: 'There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary' and 'You don't need to create new machine readable files, AI text files, or markup to appear in these features. There's also no special schema.org structured data that you need to add.' Directs owners to ordinary SEO fundamentals plus existing controls (nosnippet, noindex, Google-Extended). Directly contradicts the core sales premise of most AEO/GEO audit tools; no competitor surveyed publishes this quote."
    },
    {
      "id": "google-ai-features-trust",
      "title": "AI features and your website",
      "url": "https://developers.google.com/search/docs/appearance/ai-features",
      "publisher": "Google Search Central",
      "type": "vendor-doc",
      "verified": "2026-08-21",
      "keyFindings": "No special optimization or additional technical requirement for AI Overviews/AI Mode beyond standard indexability and snippet eligibility; no mention of on-page trust signals.",
      "accessed": "2026-08-21"
    },
    {
      "id": "google-ai-optimization-guide",
      "title": "Google's Guide to Optimizing for Generative AI Features on Google Search",
      "url": "https://developers.google.com/search/docs/fundamentals/ai-optimization-guide",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-21",
      "keyFindings": "The single most important primary source for this domain, and it is largely COUNTER-evidence for structural AEO tactics. Google defines query fan-out as 'A set of concurrent, related queries generated by the model to request more information and fetch additional relevant search results.' It recommends organizing content so that 'web pages are organized by paragraphs and sections, along with headings that provide a clear structure' and to 'Write content for your human audience.' It explicitly enumerates what is NOT needed: 'There's no requirement to break your content into tiny pieces for AI to better understand it'; 'You don't need to write in a specific way just for generative AI search'; 'You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search'; 'Structured data isn't required for generative AI search, and there's no special schema.org markup you need to add'; llms.txt — 'Google Search itself doesn't use them.' It stresses first-hand original value: 'A first-hand review provides a unique perspective based on personal experience, whereas a summary of existing content simply restates information already available elsewhere.'"
    },
    {
      "id": "google-ai-optimization-mythbusting",
      "title": "AI features and your website — AI optimization guide (mythbusting section)",
      "url": "https://developers.google.com/search/docs/fundamentals/ai-optimization-guide",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-21",
      "keyFindings": "DECISIVE VENDOR DENIAL, explicitly naming the file: \"LLMS.txt files and other 'special' markup: You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search\" and \"It's completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files. Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them.\""
    },
    {
      "id": "google-ap2-github",
      "title": "AP2 — Agent Payments Protocol",
      "url": "https://github.com/google-agentic-commerce/AP2",
      "type": "draft-spec",
      "publisher": "Google",
      "verified": "2026-08-20",
      "keyFindings": "Apache 2.0, maintained by google-agentic-commerce. Standardizes agent payment AUTHORIZATION via signed Intent/Cart/Payment mandates carried as W3C Verifiable Credentials. Confirms AP2 defines no deep links, well-known URLs, or markup-based website discovery — it is orthogonal to how an agent finds a product page."
    },
    {
      "id": "google-ard-announcement",
      "title": "Announcing the Agentic Resource Discovery specification",
      "url": "https://developers.googleblog.com/announcing-the-agentic-resource-discovery-specification",
      "type": "announcement",
      "publisher": "Google",
      "keyFindings": "ARD is a Linux Foundation working-group specification with Google, Microsoft and Hugging Face among its contributors, published 2026-06-17 under Apache 2.0.",
      "verified": "2026-08-24"
    },
    {
      "id": "google-article-author",
      "title": "Article structured data — author best practices",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/article#author-best-practices",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-21",
      "keyFindings": "Use `Person` for people and `Organization` for organizations (\"Don't use the `Thing` type\"); \"Google can understand both `sameAs` and `url` when disambiguating authors\"."
    },
    {
      "id": "google-article-structured-data",
      "title": "Article (Article, NewsArticle, BlogPosting) Structured Data",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/article",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-21",
      "keyFindings": "'There are no required properties; instead, add the properties that apply to your content' — all author and date properties are recommended, not required. author.name: specify only the author's name, no job titles, honorifics or publisher info. author.url: 'a web page that uniquely identifies the author of the article. For example, the author's social media page, an \"about me\" page, or a bio page.' sameAs is offered as an alternative to url for author disambiguation. Best practice: 'list each author in their own author field' rather than combining names; use Person or Organization type with url pointing to author profile pages. datePublished/dateModified help Google provide 'more accurate date information' and drive date display in results. This page also hosts Google's author-markup best-practices section."
    },
    {
      "id": "google-assistant-content-overview",
      "title": "Actions from web content — overview",
      "url": "https://developers.google.com/assistant/content/overview",
      "publisher": "Google Assistant developers",
      "type": "vendor-doc",
      "keyFindings": "Surviving web-content Assistant integrations are FAQs, Media, News and Podcasts. How-to guides and Recipes are no longer listed. developers.google.com/assistant/content/howto returns HTTP 404 (verified by curl).",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "google-auto-item-updates",
      "title": "Automatic item updates",
      "url": "https://support.google.com/merchants/answer/3246284",
      "type": "vendor-doc",
      "publisher": "Google Merchant Center",
      "verified": "2026-08-21",
      "keyFindings": "Google repairs feed/landing-page discrepancies 'using the structured data markup the crawlers find on your website'. Required markup: price + priceCurrency (or priceSpecification), availability with valid ItemAvailability values, itemCondition mapped to NewCondition/RefurbishedCondition/UsedCondition. When markup is missing, ML 'advanced data extractors' run instead, and 'if the extractors are unable to determine price availability, or condition information, your products will be subject to item-level disapprovals.' This makes PDP structured data an auditable proxy for feed eligibility."
    },
    {
      "id": "google-breadcrumb-structured-data",
      "title": "Breadcrumb structured data",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/breadcrumb",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-21",
      "keyFindings": "\"Google Search uses breadcrumb markup in the body of a web page to categorize the information from the page in search results.\" Breadcrumb trails display on desktop across all regions/languages, and the same page can be shown with different trails depending on the query — i.e. Google ingests the markup as a hierarchy signal, not merely a display string."
    },
    {
      "id": "google-build-sitemap",
      "title": "Build and Submit a Sitemap | Google Search Central",
      "url": "https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-21",
      "keyFindings": "Google supports XML sitemaps, RSS, mRSS, Atom 1.0 and text sitemaps: 'Google accepts RSS 2.0 and Atom 1.0 feeds.' This is the clearest primary evidence that a feed doubles as a URL-discovery surface. Caveat: 'submitting a sitemap is merely a hint: it doesn't guarantee that Google will download the sitemap or use the sitemap for crawling URLs on the site.' lastmod used 'if it's consistently and verifiably accurate'."
    },
    {
      "id": "google-ca-sunset",
      "title": "Conversational Actions sunset",
      "url": "https://developers.google.com/assistant/ca-sunset",
      "publisher": "Google Assistant developers",
      "type": "announcement",
      "keyFindings": "'Google is sunsetting Conversational Actions on June 13, 2023, which means these custom experiences for Google Assistant users will no longer be available to users or developers.'",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "google-canonicalization-docs",
      "title": "How to specify a canonical URL with rel=\"canonical\" and other methods",
      "url": "https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-21",
      "keyFindings": "rel=canonical is \"a strong signal that the specified URL should become canonical\" — a hint, not a directive. Purpose includes \"to specify which URL that you want people to see in search results.\" Contains no mention of AI features."
    },
    {
      "id": "google-common-crawlers",
      "title": "Google crawlers and fetchers (user agents) — Common crawlers",
      "url": "https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-21",
      "keyFindings": "Google-Extended is 'a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding ... in Gemini Apps and Grounding with Google Search on Vertex AI'. It 'does not impact a site's inclusion in Google Search nor is it used as a ranking signal.' Also documents Google-CloudVertexBot (owner-requested Vertex AI Agent builds) and GoogleOther (generic/R&D one-off crawls)."
    },
    {
      "id": "google-consolidate-duplicate-urls",
      "title": "How to Specify a Canonical URL with rel=\"canonical\" and Other Methods",
      "url": "https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-21",
      "keyFindings": "Canonicalization goals: 'To specify which URL that you want people to see in search results' and 'To consolidate signals for similar or duplicate pages'. Signal strength ranking: redirects = 'a strong signal'; rel=canonical = 'a strong signal that the specified URL should become canonical'; sitemaps = 'a weak signal that helps the URLs that are included in a sitemap become canonical'. Also: 'your site will likely do just fine without specifying a canonical preference.'"
    },
    {
      "id": "google-crawl-budget",
      "title": "Large site owner's guide to managing your crawl budget",
      "url": "https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget",
      "publisher": "Google",
      "type": "vendor-doc",
      "keyFindings": "Crawl capacity limit is governed by crawl health: \"If the site responds consistently and its response times (including latency and Time-to-First Byte) remain stable or improve, the limit goes up.\" Also \"If Google can load and render your pages faster, we might be able to read more content from your site.\" Does not mention resource hints, preconnect, prefetch, HTTP/2 push, or client-side subresource loading anywhere.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "google-crawl-budget-docs",
      "title": "Large site owner's guide to managing your crawl budget",
      "url": "https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-21",
      "keyFindings": "The only first-party vendor doc tying response latency to crawl volume: 'If the site slows down (latency increases or response times become longer)... the limit goes down and Google crawls less.' Crawl capacity limit = hostload. Also: 'Soft 404 pages will continue to be crawled, and waste your budget'; 'Google reads your sitemap regularly... we recommend including the <lastmod> tag'; returning 304 'tells Google to reuse the cached version, saving your server bandwidth and resources.'"
    },
    {
      "id": "google-crawling-infra-overview",
      "title": "Google Crawler (User Agent) Overview — crawling infrastructure",
      "url": "https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-20",
      "keyFindings": "Google's crawling infrastructure supports HTTP/1.1 and HTTP/2 (default HTTP/1.1; sites opt out with 421). Supports ETag and Last-Modified conditional requests, but 'Individual Google crawlers and fetchers may or may not make use of caching, depending on the needs of the product' — an important hedge for any cache-header claim."
    },
    {
      "id": "google-faqpage-deprecation",
      "title": "FAQPage structured data (deprecation notice)",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/faqpage",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-21",
      "keyFindings": "Carries a deprecation notice: \"This feature will no longer appear in Google Search results, as announced in the changelog entry in May 2026.\" Google's standing guidance is that unsupported markup can be left in place because \"other search engines and services may be using it\" — an explicit acknowledgement that Google itself no longer consumes it for display."
    },
    {
      "id": "google-faqpage-structured-data",
      "title": "FAQPage (FAQ) Structured Data Documentation",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/faqpage",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-21",
      "keyFindings": "Deprecation counter-evidence for the FAQ signal: as of September 14, 2023 the FAQ rich result 'is only shown for well-known, authoritative government and health websites.' The markup is specified as describing an FAQ section actually presented visibly on the page, not hidden data."
    },
    {
      "id": "google-featured-snippets",
      "title": "Featured Snippets and Your Website",
      "url": "https://developers.google.com/search/docs/appearance/featured-snippets",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-20",
      "keyFindings": "Counter-evidence against markup-driven extraction: asked whether you can mark content to become a featured snippet, Google answers 'You can't. Google systems determine whether a page would make a good featured snippet for a user's search request, and if so, elevates it.' The doc gives no guidance on paragraph/list/table formats or definition patterns; its only actionable lever is max-snippet length ('Featured snippets will only appear if enough text can be shown to generate a useful featured snippet')."
    },
    {
      "id": "google-fix-search-javascript",
      "title": "Fix Search-related JavaScript problems",
      "url": "https://developers.google.com/search/docs/crawling-indexing/javascript/fix-search-javascript",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-20",
      "keyFindings": "Direct quote on SPA catch-all rewrites: 'In a single-page application (SPA), this can be especially difficult. When a SPA is using client-side JavaScript to handle errors they often report a 200 HTTP status code instead of the appropriate status code. This can lead to error pages being indexed...' Recommended fixes: redirect to a URL that returns 404, or set robots meta noindex."
    },
    {
      "id": "google-gmail-one-click-action",
      "title": "One Click Actions — Gmail markup reference",
      "url": "https://developers.google.com/gmail/markup/reference/one-click-action",
      "type": "vendor-doc",
      "publisher": "Google (Gmail)",
      "verified": "2026-08-20",
      "keyFindings": "Gmail parses and EXECUTES schema.org potentialAction: \"One Click actions currently supported in Gmail are: ConfirmAction, SaveAction.\" Uses EmailMessage + potentialAction + HttpActionHandler with a url endpoint; JSON-LD and Microdata both accepted. The only verified case of a mainstream consumer actually invoking a schema.org Action — but in email, not on web pages."
    },
    {
      "id": "google-helpful-content",
      "title": "Creating Helpful, Reliable, People-First Content",
      "url": "https://developers.google.com/search/docs/fundamentals/creating-helpful-content",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-21",
      "keyFindings": "E-E-A-T status, verbatim: 'While E-E-A-T itself isn't a specific ranking factor, using a mix of factors that can identify content with good E-E-A-T is useful.' And 'Of these aspects, trust is most important. The others contribute to trust, but content doesn't necessarily have to demonstrate all of them.' Authorship ('Who'): 'Is it self-evident to your visitors who authored your content? Do pages carry a byline, where one might be expected?' and 'We strongly encourage adding accurate authorship information, such as bylines to content where readers might expect it.' Originality: 'Does the content provide original information, reporting, research, or analysis? If the content draws on other sources, does it avoid simply copying or rewriting those sources, and instead provide substantial additional value and originality?' Dates anti-pattern: 'Are you changing the date of pages to make them seem fresh when the content has not substantially changed?'"
    },
    {
      "id": "google-howto-doc-removed",
      "title": "HowTo structured data (removed documentation)",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/how-to",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-20",
      "keyFindings": "Documentation removed; the notice states the How-to structured data documentation was removed \"as this rich result is no longer shown in search results, on both desktop and mobile devices\" (14 September 2023). No successor feature named."
    },
    {
      "id": "google-howto-faq-blog",
      "title": "Changes to HowTo and FAQ rich results",
      "url": "https://developers.google.com/search/blog/2023/08/howto-faq-changes",
      "publisher": "Google Search Central Blog",
      "type": "announcement",
      "keyFindings": "Posted 2023-08-08; updated 2023-09-14: 'As of September 13, Google Search no longer shows How-to rich results on desktop, which means this result type is now deprecated.' Rich Results Test support dropped in 30 days, Search Console API in 180 days. Also: 'Structured data that's not being used does not cause problems for Search, but also has no visible effects in Google Search.'",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "google-howto-faq-changes-2023",
      "title": "Changes to HowTo and FAQ rich results",
      "url": "https://developers.google.com/search/blog/2023/08/howto-faq-changes",
      "type": "vendor-doc",
      "publisher": "Google Search Central Blog",
      "verified": "2026-08-21",
      "keyFindings": "August 2023 announcement that began the retirement: HowTo rich results limited then removed (fully gone by 14 Sept 2023, desktop and mobile), FAQ rich results restricted to well-known authoritative government and health websites — the restriction that preceded full FAQ removal in May 2026. (Page resolves; body is JS-rendered and was not extractable via fetch — content corroborated by the changelog entries in google-search-updates-changelog.)"
    },
    {
      "id": "google-howto-structured-data",
      "title": "HowTo Structured Data (Removed) — Search Central Documentation Updates",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/how-to",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-20",
      "keyFindings": "Deprecation counter-evidence for step-list markup: on September 14, 2023 Google 'Removed the How-to structured data documentation, as this rich result is no longer shown in search results, on both desktop and mobile devices.' Step-based rich results are no longer a Google consumer of procedural content markup."
    },
    {
      "id": "google-http-caching-blog",
      "title": "Crawling December: HTTP caching",
      "url": "https://developers.google.com/search/blog/2024/12/crawling-december-caching",
      "type": "vendor-doc",
      "publisher": "Google Search Central Blog",
      "verified": "2026-08-20",
      "keyFindings": "URL resolves (fetch returned the Search Central blog surface rather than the article body, so the quotes below are corroborated by the crawling-infrastructure doc). Google supports ETag/If-None-Match and Last-Modified/If-Modified-Since, recommends ETag as less error-prone, and states a 304 with no body saves compute and transfer, helping large slow-changing sites be crawled more efficiently."
    },
    {
      "id": "google-http-status",
      "title": "How HTTP status codes, and network and DNS errors affect Google Search",
      "url": "https://developers.google.com/search/docs/crawling-indexing/http-network-errors",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-21",
      "keyFindings": "304 Not Modified: 'Google crawlers signal the next processing system that the content is the same as last time it was crawled.' 5xx 'prompt Google's crawlers to temporarily slow down with crawling'. 4xx causes crawl frequency to gradually decrease. Up to 10 redirect hops followed."
    },
    {
      "id": "google-http-status-codes",
      "title": "How HTTP status codes, and network and DNS errors affect Google Search",
      "url": "https://developers.google.com/search/docs/crawling-indexing/http-network-errors",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-21",
      "keyFindings": "Defines soft 404 as content that 'suggests an error... an empty page or an error message' returned with a 2xx status. States '5xx and 429 server errors prompt Google's crawlers to temporarily slow down with crawling', with the reduction proportional to the share of erroring URLs. 304 signals unchanged content downstream."
    },
    {
      "id": "google-image-seo-docs",
      "title": "Image SEO Best Practices — Google Search Central",
      "url": "https://developers.google.com/search/docs/appearance/google-images",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-20",
      "keyFindings": "The single strongest vendor statement in this domain: 'Google uses alt text along with computer vision algorithms and the contents of the page to understand the subject matter of the image.' Also documents that Google draws image context from 'captions and image titles', and advises placing images 'near relevant text and on pages that are relevant to the image subject matter', plus descriptive filenames."
    },
    {
      "id": "google-intro-structured-data",
      "title": "Intro to how structured data markup works",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-21",
      "keyFindings": "\"Google uses structured data that it finds on the web to understand the content of the page.\" Supports JSON-LD (recommended), Microdata and RDFa. Critically for auditing: \"Google can read JSON-LD data when it is dynamically injected into the page's contents, such as by JavaScript code\" — Googlebot renders, whereas Merchant Center's crawler and the AI crawlers do not."
    },
    {
      "id": "google-js-seo",
      "title": "JavaScript SEO Basics",
      "url": "https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics",
      "publisher": "Google Search Central",
      "type": "vendor-doc",
      "verified": "2026-08-20",
      "keyFindings": "Names no JavaScript framework and never states that framework choice matters. Frames everything as rendering outcome: 'Keep in mind that server-side or pre-rendering is still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript', and 'Once Google's resources allow, a headless Chromium renders the page and executes the JavaScript.' Requirements are content in the DOM and crawlable <a href> links, not a particular stack.",
      "accessed": "2026-08-21"
    },
    {
      "id": "google-keywords-meta-blog-2009",
      "title": "Google does not use the keywords meta tag in web ranking",
      "url": "https://developers.google.com/search/blog/2009/09/google-does-not-use-keywords-meta-tag",
      "type": "vendor-doc",
      "publisher": "Google Search Central Blog (Sept 2009)",
      "verified": "2026-08-20",
      "keyFindings": "Title is the finding: Google states outright it does not use the keywords meta tag in web ranking. Still live on Google's own domain 17 years later; reinforced by the current supported-meta-tags page."
    },
    {
      "id": "google-links-crawlable",
      "title": "Make Your Links Crawlable",
      "url": "https://developers.google.com/search/docs/crawling-indexing/links-crawlable",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-20",
      "keyFindings": "'Google can only crawl your link if it's an <a> HTML element with an href attribute.' Links relying solely on JavaScript event handlers or non-standard markup are not reliably parsed. Internal-link requirement, verbatim: 'Every page you care about should have a link from at least one other page on your site.' Establishes the crawl-discovery mechanism that applies equally to any HTML crawler (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot); makes no claim about internal-link volume affecting ranking or citation weight."
    },
    {
      "id": "google-local-business-doc",
      "title": "Local business structured data",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/local-business",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-21",
      "keyFindings": "LocalBusiness markup can drive a knowledge panel, a carousel of related businesses, hours/departments/reviews display, and Maps Booking API reservation flows. Explicit caveat: \"Google does not guarantee that features that consume structured data will show up in search results.\" The page does not name Google Business Profile as an alternative source."
    },
    {
      "id": "google-local-business-sd",
      "title": "Local business (LocalBusiness) structured data",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/local-business",
      "publisher": "Google",
      "type": "vendor-doc",
      "keyFindings": "Google's documented mechanism for business contact information is JSON-LD schema.org: `address` as a `PostalAddress` with streetAddress/addressLocality/addressRegion/postalCode/addressCountry, and `telephone` for the primary contact number. The HTML <address> element is not mentioned anywhere in the documentation.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "google-localbusiness-booking",
      "title": "Local business (LocalBusiness) structured data",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/local-business",
      "publisher": "Google Search Central",
      "type": "vendor-doc",
      "keyFindings": "For reservations and orders Google directs developers to the Maps Booking API, not to ReserveAction/ConfirmAction markup: 'you can use the Maps Booking API to enable bookings, payments, and other actions.'",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "google-localbusiness-doc",
      "title": "Local business (LocalBusiness) structured data",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/local-business",
      "publisher": "Google Search Central",
      "type": "vendor-doc",
      "keyFindings": "Contains no potentialAction / OrderAction / ReserveAction guidance. Instead: 'If you want to help users to make a reservation or place an order directly in Search results, you can use the Maps Booking API to enable bookings, payments, and other actions.' Google routes actions through an API, not markup.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "google-localized-versions",
      "title": "Localized versions of your pages (hreflang)",
      "url": "https://developers.google.com/search/docs/specialty/international/localized-versions",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-20",
      "keyFindings": "Critical counter-evidence for the lang attribute: \"Google doesn't use hreflang or the HTML lang attribute to detect the language of a page; instead, we use algorithms to determine the language.\" hreflang's documented role is telling Google which localized variant to serve, not language detection."
    },
    {
      "id": "google-media-actions",
      "title": "Media Actions",
      "url": "https://developers.google.com/actions/media",
      "publisher": "Google",
      "type": "vendor-doc",
      "keyFindings": "Feed-based and partner-gated: providers submit 'a feed, a JSON object containing schema.org entities representing items in your media catalog', and 'Google is working with a limited number of providers at a time to integrate each provider into the feature.' Not page-level potentialAction consumption.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "google-merchant-automated-feeds",
      "title": "Add products automatically from your online store (automated feeds / website crawl)",
      "url": "https://support.google.com/merchants/answer/7538732",
      "type": "vendor-doc",
      "publisher": "Google Merchant Center Help",
      "verified": "2026-08-20",
      "keyFindings": "Automated feeds are built by \"website crawl\", which uses structured data plus sitemap information to extract product data from the site. Requires correctly implemented structured data markup for all required attributes, correct robots.txt, Googlebot unblocked. Google re-checks the site at least every 24 hours. Offer/GTIN/SKU annotation is what lets the crawler match page offers to catalog items."
    },
    {
      "id": "google-merchant-identifiers",
      "title": "Merchant Center — unique product identifiers",
      "url": "https://support.google.com/merchants/answer/6324461",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-21",
      "keyFindings": "\"Accurate matching to products can't be assured when GTIN is missing\"; products submitted without identifiers \"are difficult to classify and may not be eligible for all Shopping features\"."
    },
    {
      "id": "google-merchant-listing",
      "title": "Merchant listing (Product) structured data",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/merchant-listing",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-21",
      "keyFindings": "Required: name, image, offers; within offers price (or priceSpecification.price) and priceCurrency (3-letter ISO 4217). Recommended: gtin|gtin8|gtin12|gtin13|gtin14, mpn, sku, availability, priceValidUntil (ISO 8601), itemCondition, url, validFrom/validThrough, hasMerchantReturnPolicy, shippingDetails (OfferShippingDetails), aggregateRating, review, description, brand. Policies are recommended at Organization level."
    },
    {
      "id": "google-merchant-setup-structured-data",
      "title": "Set up structured data for Merchant Center",
      "url": "https://support.google.com/merchants/answer/7331077",
      "type": "vendor-doc",
      "publisher": "Google Merchant Center Help",
      "verified": "2026-08-20",
      "keyFindings": "Requires a Product object with a nested Offer object. JSON-LD recommended (\"distinct from any user-facing code, which makes it easier to maintain\"). Repeats the no-JavaScript rule. price, priceCurrency, availability, condition mandatory for automatic item updates."
    },
    {
      "id": "google-merchant-spec",
      "title": "Product data specification",
      "url": "https://support.google.com/merchants/answer/7052112",
      "type": "vendor-doc",
      "publisher": "Google Merchant Center",
      "verified": "2026-08-21",
      "keyFindings": "Required: id (max 50, unique), title (max 150), description (max 5000, 'should match landing page content'), link, image_link (500x500 min enforced 2027-01-31), availability from {in_stock,out_of_stock,preorder,backorder} and it 'must match the availability from your landing page', price which 'must match landing page and checkout prices', brand (max 70). gtin strongly recommended; mpn required when no manufacturer GTIN; condition required if used/refurbished; item_group_id required for variants in BR/FR/DE/JP/UK/US."
    },
    {
      "id": "google-merchant-structured-data",
      "title": "Merchant Center — structured data for products",
      "url": "https://support.google.com/merchants/answer/6069143",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-21",
      "keyFindings": "\"Structured data lets Google and other web platforms automatically read your site and directly pull product data from your HTML\"; markup also powers the website-crawl feed input method."
    },
    {
      "id": "google-merchant-supported-structured-data",
      "title": "Supported structured data attributes and values — Merchant Center",
      "url": "https://support.google.com/merchants/answer/6386198",
      "type": "vendor-doc",
      "publisher": "Google Merchant Center Help",
      "verified": "2026-08-20",
      "keyFindings": "Explicit schema.org → product-attribute mapping: id→sku, title→name, description→description, image_link→image, gtin→gtin8/gtin12/gtin13/gtin14/gtin/isbn, price→price+priceCurrency, availability→availability, condition→itemCondition. \"If you're using automatic item updates, make sure to specify the schema.org properties price, priceCurrency, availability, and condition.\" Hard constraint: \"Structured data markup must be present in the HTML returned from the web server. The structured data markup can't be generated with JavaScript after the page has loaded.\" JSON-LD recommended. Price without currency symbols/separators; currency as ISO 4217."
    },
    {
      "id": "google-multi-regional",
      "title": "Managing multi-regional and multilingual sites",
      "url": "https://developers.google.com/search/docs/specialty/international/managing-multi-regional-sites",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-21",
      "keyFindings": "\"We don't use any code-level language information such as `lang` attributes, or the URL... Google uses the visible content of your page\" to determine language."
    },
    {
      "id": "google-nosnippet",
      "title": "Robots meta tag, data-nosnippet, and X-Robots-Tag specifications",
      "url": "https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-21",
      "keyFindings": "data-nosnippet marks textual parts of a page as excluded from snippets across web search, Images, Discover AND AI Overviews. Valid only on <span>, <div>, <section>; boolean (any value, including 'false', means on); must be present at DOM creation, not added by JS. This is the documented consumer behavior linking a page-level marker to an AI answer surface."
    },
    {
      "id": "google-organization-schema",
      "title": "Organization (Organization) Structured Data",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/organization",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-21",
      "keyFindings": "name, url and logo help Google 'uniquely identify your organization'. Explicit consistency guidance: 'Use the same name and alternateName that you're using for your site name.' sameAs links to 'a page on another website with additional information about your organization' (social profiles, review-site listings); multiple URLs allowed. Properties can 'influence visual elements in Search results (such as which logo is shown in Search results and your knowledge panel)' — documenting Google Knowledge Panel as a consumer. The doc does not mandate cross-web name consistency, only internal consistency."
    },
    {
      "id": "google-organization-sd",
      "title": "Organization (Organization) structured data",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/organization",
      "publisher": "Google Search Central",
      "type": "vendor-doc",
      "verified": "2026-08-21",
      "keyFindings": "Recommends placing Organization markup 'on your home page, or a single page that describes your organization, for example the about us page'. Supported properties are identity/contact/registry fields (name, logo, sameAs, vatID, duns, leiCode) — there are no credential, expertise, or award properties at Organization level.",
      "accessed": "2026-08-21"
    },
    {
      "id": "google-organization-structured-data",
      "title": "Organization schema markup",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/organization",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-21",
      "keyFindings": "Organization markup helps \"disambiguate your organization in search results\" and can \"influence visual elements in Search results (such as which logo is shown in Search results and your knowledge panel)\". Recommends name/alternateName plus real-world presence (address, telephone) and online presence (url, logo). No AI-specific claims anywhere on the page."
    },
    {
      "id": "google-pagination",
      "title": "Pagination and incremental page loading",
      "url": "https://developers.google.com/search/docs/specialty/ecommerce/pagination-and-incremental-page-loading",
      "publisher": "Google Search Central",
      "type": "vendor-doc",
      "verified": "2026-08-20",
      "keyFindings": "'Google no longer uses these tags, although these links may still be used by other search engines.' Recommends instead: 'consider using a sitemap file or a Google Merchant Center feed to help Google find all of the products on your site.' Direct vendor renunciation of rel=prev/next.",
      "accessed": "2026-08-21"
    },
    {
      "id": "google-pagination-blog-2011",
      "title": "Pagination with rel=\"next\" and rel=\"prev\" (Google Search Central Blog, with deprecation banner)",
      "url": "https://developers.google.com/search/blog/2011/09/pagination-with-relnext-and-relprev",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-20",
      "keyFindings": "Carries the retraction banner verbatim: 'rel=prev/next is not an indexing signal anymore.' Google publicly confirmed the deprecation in March 2019, revealing the change had already been in effect for years."
    },
    {
      "id": "google-pagination-ecommerce",
      "title": "Ecommerce Pagination and Incremental Page Loading | Google Search Central",
      "url": "https://developers.google.com/search/docs/specialty/ecommerce/pagination-and-incremental-page-loading",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-20",
      "keyFindings": "'Google no longer uses these tags [rel=next/prev], although these links may still be used by other search engines.' Positive guidance: 'Include links from each page to the following page using <a href> tags. This can help Googlebot find subsequent pages.' On load-more buttons: \"Google's crawlers don't 'click' buttons and generally don't trigger JavaScript functions that require user actions.\""
    },
    {
      "id": "google-passage-ranking-blog",
      "title": "How AI is powering a more helpful Google (Search On 2020) — passage ranking announcement",
      "url": "https://blog.google/products/search/search-on/",
      "type": "article",
      "publisher": "Google (The Keyword blog)",
      "verified": "2026-08-20",
      "keyFindings": "Primary vendor statement that retrieval operates below page level: 'We've recently made a breakthrough in ranking and are now able to better understand the relevancy of specific passages. By understanding passages in addition to the relevancy of the overall page, we can find that needle-in-a-haystack information you're looking for. This technology will improve 7 percent of search queries across all languages as we roll it out globally.' Google's SearchLiaison later clarified that pages, not passages, remain the indexing unit and passages are 'an additional ranking factor'."
    },
    {
      "id": "google-product-snippet",
      "title": "Product snippet (structured data)",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/product-snippet",
      "type": "vendor-doc",
      "publisher": "Google",
      "keyFindings": "`ratingValue` is required and \"at least one of `ratingCount` or `reviewCount` is required\"; `Product` is among the supported host types for review snippets.",
      "verified": "2026-08-21"
    },
    {
      "id": "google-product-structured-data",
      "title": "Product structured data (Product snippets and Merchant listings)",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/product",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-20",
      "keyFindings": "\"Providing both structured data on web pages and a Merchant Center feed maximizes your eligibility to experiences and helps Google correctly understand and verify your data.\" \"Some experiences combine data from structured data and Google Merchant Center feeds if both are available. For example, product snippets may use pricing data from your merchant feed if it's not present in the structured data on the page.\" No mention of AI Mode/AI Overviews or the Shopping Graph by name."
    },
    {
      "id": "google-product-variants",
      "title": "Product variant (ProductGroup) structured data",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/product-variants",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-20",
      "keyFindings": "ProductGroup REQUIRED: name, productGroupID (parent SKU, must align with inProductGroupWithID on variants). RECOMMENDED: variesBy (schema.org URLs — color, size, suggestedAge, suggestedGender, material, pattern), hasVariant (nested variant Products, or variants point back with isVariantOf). Each variant Product must carry sku or gtin (unique per variant), offers (price + availability), and the color/size values matching variesBy."
    },
    {
      "id": "google-publication-dates",
      "title": "Add a Byline Date to Google Search Results",
      "url": "https://developers.google.com/search/docs/appearance/publication-dates",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-21",
      "keyFindings": "Documented consumer behavior for visible dates: 'Add a user-visible date to the page and feature it prominently. Label your dates appropriately with text like \"Publish\" or \"Last updated\".' Examples given include 'Published February 4, 2019' and 'Last updated: Feb 14, 2018'. 'Google doesn't depend on a single date factor because all factors can be prone to issues. That's why our systems look at several factors to determine our best estimate of when a page was published or significantly updated.' Best practices: date required / time optional; 'Make your dates and times consistent. Ensure that the date (and optional time and timezone) match between the equivalent user-visible and structured values.'; 'Don't specify future dates'; 'Minimize the presence of other dates on the page'. Also warns elsewhere against artificially freshening a page without substantive change."
    },
    {
      "id": "google-qrg-2025",
      "title": "Google Search Quality Rater Guidelines (General Guidelines), September 11, 2025",
      "url": "https://guidelines.raterhub.com/searchqualityevaluatorguidelines.pdf",
      "type": "vendor-doc",
      "publisher": "Google LLC",
      "verified": "2026-08-20",
      "keyFindings": "Verified by reading the PDF directly: title 'General Guidelines', dated September 11, 2025, Copyright 2025. Content-creator identity is structurally central to Page Quality rating — §2.5.2 'Finding Who is Responsible for the Website and Who Created the Content on the Page', §2.5.3 'Finding About Us, Contact Information, and Customer Service Information', §3.3 'Reputation of the Website and Content Creators' (incl. §3.3.2 Customer Reviews as Reputation Information, §3.3.4 Reputation of the Content Creators), §3.4 'Experience, Expertise, Authoritativeness, and Trust (E-E-A-T)', §4.5.1 'Inadequate Information about the Website or Content Creator', §4.5.2 'Lowest E-E-A-T', §5.1 'Lacking E-E-A-T', §5.5 'Unsatisfying Amount of Information about the Website or Content Creator', §7.3 'High Level of E-E-A-T', §8.3 'Very High Level of E-E-A-T'. IMPORTANT CAVEAT: this is a human-rater calibration document, not an algorithm specification, and no AI Overviews rating section appears in the table of contents I verified."
    },
    {
      "id": "google-qualify-outbound-links",
      "title": "Qualify Your Outbound Links to Google (nofollow, sponsored, ugc)",
      "url": "https://developers.google.com/search/docs/crawling-indexing/qualify-outbound-links",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-20",
      "keyFindings": "'Links marked with these rel attributes will generally not be followed. Remember that the linked pages may be found through other means, such as sitemaps or links from other sites, and thus they may still be crawled.' Confirms nofollow is a HINT, not a directive, since September 2019."
    },
    {
      "id": "google-quality-rater-guidelines",
      "title": "Search Quality Rater Guidelines (11 September 2025)",
      "url": "https://static.googleusercontent.com/media/guidelines.raterhub.com/en//searchqualityevaluatorguidelines.pdf",
      "type": "vendor-doc",
      "publisher": "Google",
      "keyFindings": "§2.5.2: \"Most websites have 'contact us' or 'about us' or 'about' pages that provide information about who owns the site.\" §3.3: \"reputation research is required for all PQ rating tasks\". Calibrates human raters; states E-E-A-T is not itself a ranking factor.",
      "verified": "2026-08-21"
    },
    {
      "id": "google-recaptcha-display",
      "title": "reCAPTCHA v2 — display",
      "url": "https://developers.google.com/recaptcha/docs/display",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-21",
      "keyFindings": "Gates the `g-recaptcha-response` token on the user submitting a successful response; without that interaction there is no token to post with the form."
    },
    {
      "id": "google-reduce-crawl",
      "title": "Reduce Googlebot crawl rate",
      "url": "https://developers.google.com/search/docs/crawling-indexing/reduce-crawl-rate",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-20",
      "keyFindings": "Checked as a candidate source for 304/conditional-request guidance: it does NOT discuss 304, If-Modified-Since or If-None-Match. Only 500/503/429 are named as crawl-rate-reducing responses. Cited here to bound the evidence for the conditional-request proposal."
    },
    {
      "id": "google-rel-ugc",
      "title": "Qualify your outbound links to Google (rel=ugc / nofollow / sponsored)",
      "url": "https://developers.google.com/search/docs/crawling-indexing/qualify-outbound-links",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-20",
      "keyFindings": "'We recommend marking user-generated content (UGC) links, such as comments and forum posts, with the ugc value.' Documents the only widely-deployed machine-readable marker distinguishing visitor-contributed content from editorial content."
    },
    {
      "id": "google-return-policy",
      "title": "Return policy (MerchantReturnPolicy) structured data",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/return-policy",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-20",
      "keyFindings": "Attachable at Organization level (hasMerchantReturnPolicy) and at Offer level. Required: EITHER applicableCountry (ISO 3166-1 alpha-2, up to 50) + returnPolicyCategory, OR merchantReturnLink. returnPolicyCategory enum: MerchantReturnFiniteReturnWindow, MerchantReturnNotPermitted, MerchantReturnUnlimitedWindow. merchantReturnDays (integer) becomes REQUIRED when category is MerchantReturnFiniteReturnWindow. Recommended: returnMethod (ReturnByMail/ReturnInStore/ReturnAtKiosk), returnFees (FreeReturn/ReturnShippingFees/ReturnFeesCustomerResponsibility), refundType (FullRefund/ExchangeRefund/StoreCreditRefund), returnShippingFeesAmount, itemDefectReturnLabelSource, returnPolicyCountry, restockingFee."
    },
    {
      "id": "google-review-snippet",
      "title": "Review Snippet (Review, AggregateRating) Structured Data",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/review-snippet",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-21",
      "keyFindings": "Supported types: Book, Course, Event, LocalBusiness, Movie, Product, Recipe, SoftwareApplication, Organization and several media types. AggregateRating requires ratingValue plus at least one of ratingCount or reviewCount, and itemReviewed.name. Review requires author (max 100 chars), itemReviewed and reviewRating.ratingValue; datePublished recommended. Critical policy constraint: structured data for reviews may not be used 'if the entity that's being reviewed controls the reviews about itself' (applies to LocalBusiness and Organization). Also prohibits fake or undisclosed incentivized reviews and aggregating reviews from other sites; marked-up review content must be visible on the page. Consumer documented is Google Search rich results — not AI Overviews or AI Mode."
    },
    {
      "id": "google-review-snippet-doc",
      "title": "Review snippet structured data",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/review-snippet",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-21",
      "keyFindings": "Supported types: Book, Course list, Event, Local business, Movie, Product, Recipe, Software App, plus CreativeWorkSeason, Episode, Game, MediaObject, MusicPlaylist, MusicRecording, Organization. Self-serving rule: \"If the entity that's being reviewed controls the reviews about itself, their pages that use LocalBusiness or any other type of Organization structured data are ineligible for star review feature.\" aggregateRating requires ratingCount or reviewCount plus ratingValue on a numeric scale (default 1–5)."
    },
    {
      "id": "google-robots-meta-tag",
      "title": "Robots meta tag, data-nosnippet, and X-Robots-Tag specifications",
      "url": "https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-21",
      "keyFindings": "Supported directives: noindex, nofollow, none, nosnippet, indexifembedded, max-snippet, max-image-preview, max-video-preview, notranslate, noimageindex, unavailable_after. Key AI statement: nosnippet \"will also prevent the content from being used as a direct input for AI Overviews and AI Mode.\" Also: \"these settings can be read and followed only if crawlers are allowed to access the pages that include these settings\" — robots.txt-blocked pages have their meta directives ignored."
    },
    {
      "id": "google-robots-txt-spec",
      "title": "Robots.txt Specifications (How Google interprets the robots.txt specification)",
      "url": "https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-20",
      "keyFindings": "BOM handling is a GOOGLE extension, not an RFC rule: 'Google ignores invalid lines in robots.txt files, including the Unicode Byte Order Mark (BOM) at the beginning of the robots.txt file, and use only valid lines.' Supported fields are user-agent, allow, disallow, sitemap — 'other fields such as crawl-delay aren't supported'. Group merging: 'If there's more than one specific group declared for a user agent, all the rules from the groups applicable to the specific user agent are combined internally into a single group.' Wildcards: '*' = 0+ valid chars, '$' = end of URL. 500 KiB size limit, content past it ignored. Sitemap line must be a fully qualified absolute URL."
    },
    {
      "id": "google-sd-policies",
      "title": "Structured data general guidelines",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/sd-policies",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-20",
      "keyFindings": "\"Don't mark up content that is not visible to readers of the page\"; \"Your structured data must be a true representation of the page content\"; hidden content is listed as a reason rich results fail. Constrains the JSON-LD bloat check: the fix is never \"delete schema\", it is \"stop shipping the entire body twice\"."
    },
    {
      "id": "google-sd-policy",
      "title": "Google Search — Structured data general guidelines",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/sd-policies",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-20",
      "keyFindings": "Verbatim: 'Don't mark up content that is not visible to readers of the page.' and 'Your structured data must be a true representation of the page content.' This is the documented-consumer-behaviour basis for auditing markup↔DOM value agreement rather than markup syntax."
    },
    {
      "id": "google-search-gallery",
      "title": "Structured data markup that Google Search supports",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/search-gallery",
      "publisher": "Google Search Central",
      "type": "vendor-doc",
      "keyFindings": "Speakable is still listed among supported features: 'Allow search engines and other applications to identify news content to read aloud on Google Assistant-enabled devices.' HowTo and sitelinks searchbox / SearchAction are NOT in the list.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "google-search-updates-changelog",
      "title": "Latest Google Search documentation updates (changelog)",
      "url": "https://developers.google.com/search/updates",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-20",
      "keyFindings": "May 8, 2026: \"Added a deprecation notice to the FAQ rich result documentation. This feature will no longer appear in Google Search starting May 7, 2026.\" June 15, 2026: \"Removed documentation for the FAQ rich result feature.\" Sept 14, 2023: \"Removed the How-to structured data documentation, as this rich result is no longer shown in search results, on both desktop and mobile devices.\" July 7, 2026: Merchant listing updates (Product.category codes; new \"Sale duration\" section using schema.org properties). May 20, 2026: hasAdultConsideration added to Merchant listing / Product variant docs — evidence that Product markup is under ACTIVE development while FAQ/HowTo are retired."
    },
    {
      "id": "google-search-updates-howto",
      "title": "Google Search documentation updates — how-to deprecation",
      "url": "https://developers.google.com/search/updates#how-to-deprecation",
      "publisher": "Google Search Central",
      "type": "vendor-doc",
      "keyFindings": "2023-09-14 entry: 'Removed the How-to structured data documentation, as this rich result is no longer shown in search results, on both desktop and mobile devices.' The former doc URL /search/docs/appearance/structured-data/how-to now 301s here (verified by curl -L). Same changelog: FAQ rich results fully removed 2026-06-15; sitelinks search box removed 2024-11-29.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "google-shopping-graph-ai-mode",
      "title": "Shopping on Google: AI Mode and virtual try-on updates from I/O 2025",
      "url": "https://blog.google/products-and-platforms/products/shopping/google-shopping-ai-mode-virtual-try-on-update/",
      "type": "article",
      "publisher": "Google (The Keyword)",
      "verified": "2026-08-20",
      "keyFindings": "\"The Shopping Graph now has more than 50 billion product listings ... each with details like reviews, prices, color options and availability ... every hour more than 2 billion of those product listings are refreshed.\" AI Mode shopping \"brings together Gemini capabilities with the Shopping Graph\"; agentic checkout adds items to cart on the merchant site. Establishes the downstream AI surface that Merchant Center (and therefore crawled Product/Offer markup) feeds. Does not itself state where listings originate."
    },
    {
      "id": "google-site-names",
      "title": "Site Names in Google Search",
      "url": "https://developers.google.com/search/docs/appearance/site-names",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-20",
      "keyFindings": "Google's ranked signals for determining a site name: (1) WebSite structured data, (2) og:site_name, (3) <title>, (4) heading elements, (5) other home page text. Verbatim consistency guidance: 'Use your site name consistently across your home page. Make sure whatever you use as the site name in structured data is consistent with how you refer to your site in other sources on your home page that our system considers.' This is the strongest documented consumer for on-page brand-name consistency — but the consumer is Google Search's site-name display, not AI citation selection."
    },
    {
      "id": "google-site-names-docs",
      "title": "Site names in Google Search",
      "url": "https://developers.google.com/search/docs/appearance/site-names",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-20",
      "keyFindings": "\"Our site name system will also consider content in og:site_name, <title>, heading elements, and other text on a home page. However, WebSite structured data is most important.\" Documents og:site_name as a real (secondary) consumed signal."
    },
    {
      "id": "google-sitelinks-searchbox-farewell",
      "title": "Farewell, Sitelinks Search Box",
      "url": "https://developers.google.com/search/blog/2024/10/sitelinks-search-box",
      "type": "vendor-doc",
      "publisher": "Google Search Central Blog",
      "verified": "2026-08-21",
      "keyFindings": "October 2024 announcement retiring the Sitelinks Search Box globally on 21 November 2024 due to declining usage — this was the only mainstream consumer of WebSite/SearchAction potentialAction markup. Search Console report and Rich Results Test highlighting removed. Google states the markup need not be removed since unsupported structured data causes no problems. (URL resolves; body is JS-rendered and was not extractable via fetch.)"
    },
    {
      "id": "google-sitelinks-searchbox-removal",
      "title": "Google Search documentation updates — sitelinks search box removed",
      "url": "https://developers.google.com/search/updates#bye-sitelinkbox",
      "publisher": "Google Search Central",
      "type": "vendor-doc",
      "keyFindings": "2024-11-29: 'Removed sitelinks search box documentation and archived the nositelinkssearchbox rule' — 'The sitelinks search box feature is no longer available in Google Search results.' The former doc URL 301s to this anchor. This was the only web-scale consumer of potentialAction.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "google-sitemap-build",
      "title": "Build and submit a sitemap",
      "url": "https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-21",
      "keyFindings": "Direct quote: 'Google uses the <lastmod> value if it's consistently and verifiably (for example by comparing to the last modification of the page) accurate.' lastmod 'should reflect the date and time of the last significant update to the page… an update to the copyright date is not [significant].' priority and changefreq are ignored. 50MB uncompressed / 50,000 URL limit per sitemap file."
    },
    {
      "id": "google-sitemap-formats",
      "title": "Build and submit a sitemap (accepted formats)",
      "url": "https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-21",
      "keyFindings": "\"Google accepts RSS 2.0 and Atom 1.0 feeds\" as a sitemap format, with the caveat that \"this feed only provides information on recent URLs.\" Important nuance: this covers a feed submitted/referenced as a sitemap — it does NOT document consumption of a <link rel=\"alternate\" type=\"application/rss+xml\"> autodiscovery tag in the head."
    },
    {
      "id": "google-snippet-docs",
      "title": "Control Your Snippets in Search Results",
      "url": "https://developers.google.com/search/docs/appearance/snippet",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-21",
      "keyFindings": "'Snippets are automatically created from page content. Snippets are designed to emphasize and preview the page content that best relates to a user's specific search.' On meta descriptions: 'Google sometimes uses the meta description HTML element if it might give users a more accurate description of the page than content taken directly from the page.' Best practices: 'Create unique descriptions for each page on your site. Identical or similar descriptions on every page of a site aren't helpful'; descriptions should be human-readable, not 'long strings of keywords'. Documents nosnippet, max-snippet:[number], data-nosnippet. Note: all of this is scoped to Search snippets — no claim is made that meta description drives AI answer selection."
    },
    {
      "id": "google-spam",
      "title": "Spam policies for Google web search — cloaking, hidden text and links",
      "url": "https://developers.google.com/search/docs/essentials/spam-policies",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-20",
      "keyFindings": "Cloaking = 'presenting different content to users and search engines'. Hidden text/links = 'placing content on a page in a way solely to manipulate search engines and not to be easily viewable by human visitors', with an enumerated technique list: white text on white background, text behind images, CSS off-screen positioning, font size or opacity set to 0, single-character links. Also names the legitimate exceptions (accordions, tabs, sliders, tooltips, screen-reader-only text) — which is exactly the false-positive allowlist a detector needs."
    },
    {
      "id": "google-speakable-doc",
      "title": "Speakable structured data (beta)",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/speakable",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-24",
      "keyFindings": "Still labelled \"This feature is in beta and subject to change\" and \"We're currently developing this feature and you may see changes in requirements or guidelines.\" Scope: \"The speakable property works for users in the U.S. that have Google Home devices set to English.\" Named consumer is Google Assistant, which \"returns up to three articles from around the web and supports audio playback using TTS for sections in the article with speakable structured data.\" News content only. No deprecation notice as of this research."
    },
    {
      "id": "google-special-tags",
      "title": "Meta tags and HTML attributes that Google supports",
      "url": "https://developers.google.com/search/docs/crawling-indexing/special-tags",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-21",
      "keyFindings": "Exhaustive list of head-level meta tags Google supports: description, robots/googlebot, notranslate, nopagereadaloud, google-site-verification, Content-Type/charset, refresh, viewport, rating. Explicitly notes Google ignores unsupported tags including the meta-keywords tag. Notably absent from the supported list: author, keywords, noai, noimageai, tdm-reservation."
    },
    {
      "id": "google-title-link-docs",
      "title": "Control your title links in search results",
      "url": "https://developers.google.com/search/docs/appearance/title-link",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-21",
      "keyFindings": "Explicitly lists the sources Google uses to generate a title link: \"Content in <title> elements\", \"Main visual title shown on the page\", \"Heading elements, such as <h1> elements\", \"Content in og:title meta tags\", anchor text, text within links pointing to the page, and WebSite structured data. This is the only major-vendor documentation that names an Open Graph property as an input to a Google Search surface."
    },
    {
      "id": "google-url-structure",
      "title": "URL Structure Best Practices for Google",
      "url": "https://developers.google.com/search/docs/crawling-indexing/url-structure",
      "type": "vendor-doc",
      "publisher": "Google Search Central",
      "verified": "2026-08-20",
      "keyFindings": "'When possible, use readable words rather than long ID numbers in your URLs.' 'Use words in your audience's language in the URL (and, if applicable, transliterated words).' 'We recommend using hyphens (-) instead of underscores (_) to separate words in your URLs.' Crucially, the page frames these as best practices for crawlability and human/machine understanding — it makes NO claim that descriptive URLs are a ranking factor, only that they help 'Google Search (and your users) better understand your site'."
    },
    {
      "id": "greshake-ipi",
      "title": "Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection",
      "url": "https://arxiv.org/abs/2302.12173",
      "type": "study",
      "publisher": "arXiv / ACM AISec",
      "verified": "2026-08-20",
      "keyFindings": "Foundational indirect prompt injection paper: adversaries 'remotely exploit LLM-integrated applications by strategically injecting prompts into data likely to be retrieved.' Demonstrated against Bing Chat (GPT-4) and code-completion engines. Establishes retrieved web content as the threat channel."
    },
    {
      "id": "growth-memo-2026-research",
      "title": "2026 Growth Memo research summary",
      "url": "https://www.growth-memo.com/p/2026-growth-memo-research-summary",
      "type": "dataset",
      "publisher": "Kevin Indig, Growth Memo",
      "verified": "2026-08-20",
      "keyFindings": "Primary publisher of the citation-position dataset (page is partially paywalled but resolves). Reiterates 44.2% of citations from the first 30% of a page, 31.1% from 30–70%, and that content buried deep in a long post is roughly 2.5x less likely to be cited. Separate 2026 study spans 3,981 domains, 115 prompts, 14 countries and 4 AI search engines (ChatGPT, AI Overviews, Gemini, AI Mode) via the Semrush AI Visibility Toolkit; documents large engine-to-engine divergence (Gemini names brands in 83.7% of appearances but links only 21.4% of the time; ChatGPT cites 87.0% of the time but names brands in 20.7%)."
    },
    {
      "id": "hcaptcha-docs",
      "title": "hCaptcha documentation",
      "url": "https://docs.hcaptcha.com",
      "type": "vendor-doc",
      "publisher": "hCaptcha",
      "verified": "2026-08-21",
      "keyFindings": "States it \"helps to protect your sites and apps from bots, spam, and other automated abuse\" — its purpose is to deny non-human clients."
    },
    {
      "id": "hf-blog-ard-launch",
      "title": "Agentic Resource Discovery launch",
      "url": "https://huggingface.co/blog/agentic-resource-discovery-launch",
      "publisher": "Hugging Face",
      "type": "announcement",
      "keyFindings": "Published 2026-06-17. Describes ai-catalog.json as the static manifest format hosted at a well-known URL; contributors from Microsoft, Google, GoDaddy, Hugging Face. Hub-side support for static manifests listed as a next step.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "hf-discover",
      "title": "huggingface/hf-discover",
      "url": "https://github.com/huggingface/hf-discover",
      "type": "repo",
      "publisher": "Hugging Face",
      "keyFindings": "First-party ARD client. `navigate()` dereferences `entries[].url` into nested catalogs and federated registries, and indexes description, tags, capabilities and representativeQueries.",
      "verified": "2026-08-24"
    },
    {
      "id": "hf-discover-fields",
      "title": "hf-discover navigation.py — fields consumed",
      "url": "https://raw.githubusercontent.com/huggingface/hf-discover/main/src/discover/navigation.py",
      "publisher": "Hugging Face",
      "type": "repo",
      "keyFindings": "Reads entries[]; _entry_haystack() matches on displayName, description, tags, capabilities, representativeQueries; uses type for media-type routing and url for traversal; identifier for dedup. Does not read specVersion or host.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "hf-discover-navigation",
      "title": "hf-discover navigation.py (ARD client)",
      "url": "https://raw.githubusercontent.com/huggingface/hf-discover/main/src/discover/navigation.py",
      "publisher": "Hugging Face",
      "type": "repo",
      "keyFindings": "well_known_catalog_url() builds {scheme}://{netloc}/.well-known/ai-catalog.json; navigate() seeds a catalog queue with it and reads entries[], following nested catalogs/registries. Only the well-known path is used — no HTML link or Link-header parsing.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "hf-discover-no-link-parsing",
      "title": "hf-discover navigation.py — no link-tag parsing",
      "url": "https://raw.githubusercontent.com/huggingface/hf-discover/main/src/discover/navigation.py",
      "publisher": "Hugging Face",
      "type": "repo",
      "keyFindings": "Counter-evidence: the only vendor client found resolves only the well-known path and has 'no logic for discovering catalog locations through HTML link elements or HTTP headers'.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "hf-discover-traversal",
      "title": "hf-discover navigation.py — URL traversal",
      "url": "https://raw.githubusercontent.com/huggingface/hf-discover/main/src/discover/navigation.py",
      "publisher": "Hugging Face",
      "type": "repo",
      "keyFindings": "entry.url is used to traverse nested catalogs and registries based on entry.type media type; dead URLs cut off federated discovery.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "html-aam-img-empty-alt",
      "title": "HTML Accessibility API Mappings 1.0 — §3.5.57 img element",
      "url": "https://www.w3.org/TR/html-aam-1.0/",
      "publisher": "W3C",
      "type": "spec",
      "keyFindings": "An img element with an empty alt attribute maps to role 'none or presentation'. The only exception: if such an img is given an accessible name via another valid naming mechanism, user agents expose it with its implicit image role. Normative proof that adding role=\"presentation\" on top of alt=\"\" is redundant.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "htmldate-docs",
      "title": "htmldate documentation",
      "url": "https://htmldate.readthedocs.io/en/latest",
      "type": "repo",
      "publisher": "htmldate",
      "verified": "2026-08-21",
      "keyFindings": "Identifies \"original and updated publication dates\" from \"`link` and `meta` elements including Open Graph protocol attributes\" — the same fields, in the same priority order, this audit reads."
    },
    {
      "id": "htmldate-extractors",
      "title": "htmldate extractors.py (date XPath constants)",
      "url": "https://raw.githubusercontent.com/adbar/htmldate/master/htmldate/extractors.py",
      "type": "vendor-doc",
      "publisher": "Adrien Barbaresi / htmldate",
      "verified": "2026-08-20",
      "keyFindings": "FAST_PREPEND includes './/*[self::div or self::h2 or self::h3 or self::h4 or self::li or self::p or self::span or self::time or self::ul]' — the <time> element is an explicit extraction target. DATE_EXPRESSIONS additionally match @itemprop and date-ish @class/@id. Notably there is no XPath keyed on the @datetime or @pubdate attribute itself, so the element is used as a text container more than as a machine-readable value carrier."
    },
    {
      "id": "htmldate-joss-paper",
      "title": "htmldate: A Python package to extract publication dates from web pages",
      "url": "https://joss.theoj.org/papers/10.21105/joss.02439",
      "type": "study",
      "publisher": "Journal of Open Source Software (Barbaresi, 2020, 5(51):2439)",
      "verified": "2026-08-20",
      "keyFindings": "Peer-reviewed provenance for the date-extraction library that trafilatura uses for metadata. 'Markup-based extraction is multilingual by nature'; the package is described as used in production on millions of documents. Establishes that <time>-bearing markup is consumed by real corpus-construction pipelines, not just by browsers."
    },
    {
      "id": "hubspot",
      "title": "HubSpot AI Search Grader",
      "url": "https://www.hubspot.com/ai-search-grader",
      "type": "vendor-doc",
      "publisher": "HubSpot",
      "verified": "2026-08-20",
      "keyFindings": "Scores 100 points across 5 dimensions: Sentiment Results (40), Presence Quality (20), Brand Recognition (20), Share of Voice (10), Market Competition (10). Queries ChatGPT/Perplexity/Gemini about the brand. Performs no crawl of the site at all — no robots.txt, llms.txt or schema inspection."
    },
    {
      "id": "hubspot-ai-search-grader",
      "title": "HubSpot AI Search Grader",
      "url": "https://www.hubspot.com/ai-search-grader",
      "type": "vendor-doc",
      "publisher": "HubSpot",
      "verified": "2026-08-20",
      "keyFindings": "The only mainstream competitor that publishes its scoring weights: composite 0-100 from Sentiment Results 40, Presence Quality 20, Brand Recognition 20, Share of Voice 10, Market Competition 10. Inputs are company name / location / product / industry — it never fetches the site, so it is brand-perception grading, not technical auditing. Weights are asserted with no justification or external source (contrast Lighthouse sourcing a11y weights to axe impact). Carries the disclaimer that recommendations 'are generated by an AI tool' and are 'no substitute for professional advice'."
    },
    {
      "id": "iana-link-relations",
      "title": "IANA Link Relation Types registry",
      "url": "https://www.iana.org/assignments/link-relations/link-relations-1.csv",
      "type": "spec",
      "publisher": "IANA",
      "verified": "2026-08-20",
      "keyFindings": "`api-catalog` link relation IS registered (RFC 9727): 'Refers to a list of APIs available from the Publisher of the link context.' `service-desc` IS registered. There is NO registered `mcp`, `mcp-server`, `agent`, `webmcp`, or `ai-plugin` link relation — so an HTML `<link rel=\"mcp\">` is an unregistered, undefined relation type."
    },
    {
      "id": "iana-well-known-linkrel",
      "title": "Well-Known URIs registry (context for link relation status)",
      "url": "https://www.iana.org/assignments/well-known-uris/well-known-uris.xhtml",
      "publisher": "IANA",
      "type": "spec",
      "keyFindings": "agent-card.json and api-catalog are registered; ai-catalog is not — the ai-catalog relation/path remains pre-registration.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "iana-well-known-registry",
      "title": "IANA Well-Known URIs Registry",
      "url": "https://www.iana.org/assignments/well-known-uris/well-known-uris.xhtml",
      "publisher": "IANA",
      "type": "spec",
      "keyFindings": "Registry lists 180+ registered suffixes (acme-challenge through xregistry) with permanent/provisional/deprecated/obsoleted statuses. 'agents.json' and 'agent' are entirely absent — the path has no reserved standing.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "iana-well-known-registry-aiplugin",
      "title": "IANA Well-Known URIs Registry (checked for ai-plugin)",
      "url": "https://www.iana.org/assignments/well-known-uris/well-known-uris.xhtml",
      "publisher": "IANA",
      "type": "spec",
      "keyFindings": "'ai-plugin' and 'ai-plugin.json' are absent from the 180+ registered well-known suffixes — the path was never standardized beyond OpenAI's own vendor convention, so nothing outlived the program's shutdown.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "iana-well-known-registry-mcp",
      "title": "IANA Well-Known URIs Registry (checked for mcp)",
      "url": "https://www.iana.org/assignments/well-known-uris/well-known-uris.xhtml",
      "publisher": "IANA",
      "type": "spec",
      "keyFindings": "'mcp' and 'mcp.json' are absent from the 180+ registered well-known suffixes, consistent with SEP-2127 still being an unmerged draft.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "iana-well-known-uris",
      "title": "IANA Well-Known URIs registry",
      "url": "https://www.iana.org/assignments/well-known-uris/well-known-uris.xhtml",
      "type": "spec",
      "publisher": "IANA",
      "verified": "2026-08-20",
      "keyFindings": "152 registered suffixes. REGISTERED: `agent-card.json` (reference https://a2a-protocol.org/latest/specification/, status permanent, change controller Linux Foundation, registered 2025-08-01); `api-catalog` (RFC 9727, permanent, IETF, 2024-12-23); `oauth-protected-resource` (RFC 9728 §3, permanent, IETF, 2024-10-22). NOT REGISTERED: mcp, mcp.json, mcp-server, ai-catalog.json, ai-plugin.json, openapi, webmcp, nlweb, llms.txt."
    },
    {
      "id": "ietf-aipref-attach",
      "title": "draft-ietf-aipref-attach: Attaching AI Preferences to Content",
      "url": "https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/",
      "publisher": "IETF AI Preferences WG",
      "type": "spec",
      "keyFindings": "Defines two mechanisms only — a Content-Usage HTTP header and a Content-Usage robots.txt directive. Explicitly: 'This document, however, does not define any specific means of embedding preferences in content.'",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "ietf-aipref-attach-draft",
      "title": "Associating AI Usage Preferences with Content in HTTP (draft-ietf-aipref-attach)",
      "url": "https://ietf-wg-aipref.github.io/drafts/draft-ietf-aipref-attach.html",
      "type": "draft-spec",
      "publisher": "IETF AI Preferences (AIPREF) WG — G. Illyes (Google), M. Thomson (Mozilla)",
      "verified": "2026-08-20",
      "keyFindings": "Editor's copy dated 19 Aug 2026, Standards Track, updates RFC 9309. DECISIVE COUNTER-EVIDENCE for HTML-head AI declarations: the draft defines exactly two attachment mechanisms — the Content-Usage HTTP response header field and a Content-Usage rule in robots.txt. It does NOT define any HTML meta element or link relation. 'Embedded Preferences' is acknowledged as out of scope, noting different methods would be needed per content type."
    },
    {
      "id": "ietf-aipref-vocab-draft",
      "title": "A Vocabulary For Expressing AI Usage Preferences (draft-ietf-aipref-vocab)",
      "url": "https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/",
      "type": "draft-spec",
      "publisher": "IETF AI Preferences (AIPREF) WG",
      "verified": "2026-08-20",
      "keyFindings": "Active WG draft (rev -06, Apr 2026). Standardizes the preference vocabulary (train-ai, train-genai, ai-input/search, etc.) that AI opt-out signals will use. Confirms the IETF trajectory for AI-usage declarations runs through HTTP headers and robots.txt, not the HTML head — meaning noai-style head meta tags are a parallel, non-standardized track."
    },
    {
      "id": "ietf-aipref-wg",
      "title": "IETF AI Preferences (aipref) working group",
      "url": "https://datatracker.ietf.org/wg/aipref/about",
      "type": "spec",
      "publisher": "IETF",
      "verified": "2026-08-21",
      "keyFindings": "The standards-track venue where TDM-REP and robots.txt AI extensions are being reconciled."
    },
    {
      "id": "ietf-draft-serra-mcp-discovery-uri",
      "title": "draft-serra-mcp-discovery-uri-04 — The \"mcp\" URI Scheme and MCP Server Discovery Mechanism",
      "url": "https://datatracker.ietf.org/doc/draft-serra-mcp-discovery-uri/",
      "type": "draft-spec",
      "publisher": "IETF (individual submission)",
      "verified": "2026-08-20",
      "keyFindings": "Version -04, updated 2026-03-26, expires 2026-09-25. Individual Internet-Draft with NO IETF stream assignment or WG adoption. Defines `mcp://example.com` URI scheme, `_mcp.{host} IN TXT` DNS discovery, and 'The client MUST perform an HTTP GET request to: https://{host}/.well-known/mcp-server'. This path conflicts with the MCP project's own chosen direction (`/.well-known/ai-catalog.json`)."
    },
    {
      "id": "ietf-web-bot-auth-architecture",
      "title": "HTTP Message Signatures for automated traffic Architecture (draft-meunier-web-bot-auth-architecture)",
      "url": "https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/",
      "type": "draft-spec",
      "publisher": "IETF (individual submission, Thibault Meunier / Cloudflare)",
      "verified": "2026-08-20",
      "keyFindings": "Revision -05, last updated 2026-03-02. Individual Internet-Draft, expired/archived, explicitly 'not endorsed by the IETF' with 'no formal standing in the IETF standards process'; superseded by draft-meunier-webbotauth-httpsig-protocol. Argues IP allow-listing, User-Agent strings and shared API keys are inadequate for identifying automated traffic."
    },
    {
      "id": "igaming-notability",
      "title": "Algorithmic Trust and Compliance: Benchmarking Brand Notability for UK iGaming Entities",
      "url": "https://arxiv.org/abs/2603.12282",
      "publisher": "arXiv",
      "type": "study",
      "verified": "2026-08-21",
      "keyFindings": "'AI Search exhibits a systematic and overwhelming bias towards Earned media (third-party, authoritative sources) over Brand-owned content'; ~78% of citations go to earned media. Compliance/regulatory signals act as authority multipliers, but derive force from third-party validation rather than self-hosted claims.",
      "accessed": "2026-08-21"
    },
    {
      "id": "imperva-bad-bot-report-2025",
      "title": "2025 Imperva Bad Bot Report",
      "url": "https://www.imperva.com/resources/resource-library/reports/2025-bad-bot-report/",
      "type": "study",
      "publisher": "Imperva / Thales",
      "verified": "2026-08-20",
      "keyFindings": "Automated traffic reached 51% of all web traffic, surpassing human traffic for the first time in a decade; 'bad bots' specifically are 37% of all internet traffic. Reports AI lowering the barrier to entry for bot operators and driving both volume and sophistication. Public page does not disclose sample construction or bot-classification methodology — a material limitation."
    },
    {
      "id": "implicator-chatgpt-499-report",
      "title": "ChatGPT Search Abandons Slow Sites With 499 Timeout Errors",
      "url": "https://www.implicator.ai/is-chatgpt-quietly-reshaping-the-web-by-penalizing-slow-sites/",
      "type": "article",
      "publisher": "Implicator.ai (reporting research by Jérôme Salomon with Oncrawl data)",
      "verified": "2026-08-20",
      "keyFindings": "Reports that OAI-SearchBot / ChatGPT-User / GPTBot requests appear as HTTP 499 (client closed request) in origin logs when the origin is too slow, and that 499 growth tracks AI-visibility decline. Attribution is to original research by Jérôme Salomon with Oncrawl data, but the article publishes NO sample size, NO timeout threshold in seconds, and NO link to an underlying methodology — treat as directional, not probative."
    },
    {
      "id": "incumbent-advantage",
      "title": "Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems",
      "url": "https://arxiv.org/abs/2606.17443",
      "publisher": "arXiv",
      "type": "study",
      "verified": "2026-08-20",
      "keyFindings": "Well-known brands receive a 100% recommendation rate when products are identical; 'authority-style marketing language, including fabricated clinical-evidence claims' breaks that monopoly with a Bias Surplus Value of +0.17 rating points, varying by model (GPT-4o-mini, Claude Sonnet, Gemini 3 Flash). Framed as a manipulation/bias vulnerability, not a best practice.",
      "accessed": "2026-08-21"
    },
    {
      "id": "indexnow-doc",
      "title": "IndexNow Protocol Documentation",
      "url": "https://www.indexnow.org/documentation",
      "type": "spec",
      "publisher": "IndexNow (Microsoft/Yandex)",
      "verified": "2026-08-20",
      "keyFindings": "Ownership is proven by hosting a UTF-8 text file at the host root named {key}.txt whose body is the key. Key must be 8-128 chars from [a-zA-Z0-9-]. Verification is a byte comparison: HTTP 403 is returned when the key is 'not found in the key file' or invalid; 422 on host/schema mismatch; 429 on rate limit; 202 means 'key validation pending'. keyLocation restricts submittable URLs to the key file's directory and deeper. Batch POST accepts up to 10,000 URLs."
    },
    {
      "id": "indexnow-faq",
      "title": "IndexNow FAQ — participating search engines",
      "url": "https://www.indexnow.org/faq",
      "type": "vendor-doc",
      "publisher": "IndexNow",
      "verified": "2026-08-20",
      "keyFindings": "Participating engines: Amazon, Bing, Naver, Seznam.cz, Yandex, Yep. Submissions to the global endpoint are shared with all participants. States IndexNow 'helps keep your content current in AI-powered search results' but names no specific LLM/Copilot consumer — the AI-consumer link is a vendor claim, not a documented pipeline."
    },
    {
      "id": "indig-chatgpt-citation-study-sel",
      "title": "44% of ChatGPT citations come from the first third of content: Study",
      "url": "https://searchengineland.com/chatgpt-citations-content-study-469483",
      "type": "study",
      "publisher": "Search Engine Land (reporting Kevin Indig / Growth Memo research)",
      "verified": "2026-08-21",
      "keyFindings": "Largest observational dataset located for passage-position and passage-linguistics. Method: 3 million ChatGPT responses and 30 million citations, narrowed to 18,012 verified citations; sentence-transformer embeddings matched answer text to source sentences. Findings: '44.2% of citations come from the first 30% of content', middle third 31.1%, final third 24.7%. Within paragraphs, 53% of cited sentences came from the MIDDLE of the paragraph, 24.5% first sentences, 22.5% last sentences. '78.4% of citations tied to questions came from headings', suggesting H2s function as prompts and the following paragraph as the answer. Cited passages showed entity richness ~20.6% proper nouns (vs typical 5–8%); definitive 'X is' phrasing cited nearly twice as often; conversational Q&A structure 2x more likely to be cited; Flesch-Kincaid grade 16 vs 19.1 for lower performers. All correlational, single-engine, single-vendor tooling."
    },
    {
      "id": "iptc-synthetic-media-guidance",
      "title": "IPTC publishes metadata guidance for AI-generated \"synthetic media\"",
      "url": "https://iptc.org/news/iptc-publishes-metadata-guidance-for-ai-generated-synthetic-media/",
      "type": "spec",
      "publisher": "IPTC",
      "verified": "2026-08-20",
      "keyFindings": "COUNTER-EVIDENCE for HTML-head AI-generated declarations. IPTC's Digital Source Type vocabulary (trainedAlgorithmicMedia, cv.iptc.org/newscodes/digitalsourcetype) is recommended for the XMP data packet embedded in image/video files, or inside a C2PA manifest. The AI-provenance standards ecosystem targets embedded file metadata and C2PA — there is no IPTC/C2PA HTML head meta tag for declaring page-level AI-generated content."
    },
    {
      "id": "ipullrank-page-speed-ai",
      "title": "Quick Tip: How Page Speed Impacts ChatGPT and Perplexity Visibility",
      "url": "https://ipullrank.com/page-speed-impacts",
      "type": "article",
      "publisher": "iPullRank (reporting Profound data)",
      "verified": "2026-08-20",
      "keyFindings": "Cites Profound analysis of a random sample of 700K pages over a multi-day period in April 2026: pages with fetch-failure rates above 75% received roughly 18x fewer citation events than more stable pages, many with zero citations. One client case: 22% AI-search visibility increase after fixing 499s. LIMITATION acknowledged in the piece: no published methodology for defining failure rates, and no explicit timeout threshold in seconds for any AI crawler."
    },
    {
      "id": "isitagentready",
      "title": "Is It Agent Ready?",
      "url": "https://isitagentready.com/",
      "type": "vendor-doc",
      "publisher": "isitagentready.com",
      "verified": "2026-08-20",
      "keyFindings": "The closest hosted analogue to Agent Lighthouse in scope: scans five categories — discoverability (robots.txt, sitemaps, DNS-AID), content accessibility (Markdown content negotiation), bot access control (AI bot rules, Content Signals, Web Bot Auth), protocol discovery (MCP, OAuth, Agent Skills, WebMCP) and commerce standards (x402, MPP, UCP, ACP). It scores, but publishes no weight table, no per-check evidence, and no consumer list; the recommendations carry an 'AI-generated recommendations. AI can make mistakes' disclaimer. Not open source. Notably it scores several signals that are pre-adoption drafts without flagging them as such."
    },
    {
      "id": "jina-reader",
      "title": "Jina Reader (r.jina.ai)",
      "url": "https://jina.ai/reader/",
      "type": "vendor-doc",
      "publisher": "Jina AI",
      "verified": "2026-08-20",
      "keyFindings": "Converts URLs to \"clean, LLM-ready\" markdown because \"raw HTML is cluttered with extraneous elements\". Documents X-Target-Selector (\"Only extract content matching these CSS selectors\"), X-Remove-Selector (\"Remove these elements before extraction\"), X-Retain-Images (\"Strip all images from the output\" to reduce token usage), X-Return-Format, X-With-Images-Summary. Token cost of images/boilerplate is an explicit product knob."
    },
    {
      "id": "jina-reader-readme",
      "title": "Jina Reader (r.jina.ai) README",
      "url": "https://raw.githubusercontent.com/jina-ai/reader/main/README.md",
      "publisher": "Jina AI",
      "type": "repo",
      "keyFindings": "An LLM/agent-facing URL-to-markdown service ('Your LLMs deserve better input'). Documents `x-respond-with: markdown` as returning markdown 'without going through readability', establishing that Readability is the default main-content pipeline for agent-consumed page text.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "jina-reader-repo",
      "title": "jina-ai/reader",
      "url": "https://github.com/jina-ai/reader",
      "type": "repo",
      "publisher": "Jina AI",
      "verified": "2026-08-21",
      "keyFindings": "HTML-to-markdown service built for LLM consumption. Uses Turndown as its conversion engine and exposes `x-md-*` headers to \"fine-tune markdown output\"."
    },
    {
      "id": "kai-spriestersbach-llmstxt-dud",
      "title": "The llms.txt is dead. More precisely: a dud.",
      "url": "https://medium.com/@kaispriestersbach/the-llms-txt-is-dead-more-precisely-a-dud-ab7bee4f469c",
      "type": "article",
      "publisher": "Kai Spriestersbach (Medium)",
      "verified": "2026-08-20",
      "keyFindings": "Widely-cited skeptical analysis arguing llms.txt has no consumers. Marked unverified: WebFetch returned HTTP 403 (Medium paywall/bot block), so I could not read the primary text and am not relying on its specific numbers."
    },
    {
      "id": "knownagents-anthropic-ai",
      "title": "anthropic-ai — Known Agents",
      "url": "https://knownagents.com/agents/anthropic-ai",
      "type": "dataset",
      "publisher": "Known Agents",
      "verified": "2026-08-20",
      "keyFindings": "Classified as an 'Undocumented AI Agent' — 'Crawls websites without disclosing its purpose, collecting data for an unknown AI use case.' 16% of top websites block anthropic-ai despite it appearing in no current Anthropic documentation. Claude-Web is likewise listed as Undocumented: 'currently unclear exactly what it's used for, since there's no official documentation.'"
    },
    {
      "id": "knownagents-bravebot",
      "title": "Bravebot — Known Agents",
      "url": "https://knownagents.com/agents/bravebot",
      "type": "dataset",
      "publisher": "Known Agents",
      "verified": "2026-08-20",
      "keyFindings": "Lists UA 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Bravebot/1.0; +https://search.brave.com/help/brave-search-crawler) Chrome/W.X.Y.Z Safari/537.36' and only 2% top-site blocking as of 2026-08-19. Directly conflicts with brave-search-crawler, where Brave states its crawler does not advertise a differentiated user agent."
    },
    {
      "id": "knownagents-bytespider",
      "title": "Bytespider — Known Agents",
      "url": "https://knownagents.com/agents/bytespider",
      "type": "dataset",
      "publisher": "Known Agents",
      "verified": "2026-08-20",
      "keyFindings": "Operated by ByteDance; typed AI Data Scraper. UA: 'Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://bytedance.sg.larkoffice.com/docx/...)'. As of 2026-08-19, '19% of top websites are blocking Bytespider'; crawls from Singapore; makes 'broad, high-volume sweeps'. Only operator-side reference is the Chinese-language zhanzhang.toutiao.com portal — there is no English vendor bot page comparable to OpenAI's or Anthropic's."
    },
    {
      "id": "knownagents-claudebot",
      "title": "ClaudeBot — Known Agents",
      "url": "https://knownagents.com/agents/claudebot",
      "type": "dataset",
      "publisher": "Known Agents",
      "verified": "2026-08-20",
      "keyFindings": "As of 2026-08-19: '21% of top websites are blocking ClaudeBot'; AI data scrapers overall generated 1.4% of all web traffic. UA: 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)'."
    },
    {
      "id": "knownagents-cohere-training-crawler",
      "title": "cohere-training-data-crawler — Known Agents",
      "url": "https://knownagents.com/agents/cohere-training-data-crawler",
      "type": "dataset",
      "publisher": "Known Agents",
      "verified": "2026-08-20",
      "keyFindings": "Operated by Cohere; typed AI Data Scraper; token is the literal string 'cohere-training-data-crawler'. 7% of top websites block it as of 2026-08-19. Separately, 'cohere-ai' is catalogued as an 'Undocumented AI Agent' — 'an unconfirmed agent possibly dispatched by Cohere's AI chat products'. No public Cohere vendor bot documentation page was located."
    },
    {
      "id": "knownagents-diffbot",
      "title": "Diffbot — Known Agents",
      "url": "https://knownagents.com/agents/diffbot",
      "type": "dataset",
      "publisher": "Known Agents",
      "verified": "2026-08-20",
      "keyFindings": "Typed AI Data Provider ('Crawls websites to supply structured content to AI systems as a third-party service'). UA: 'Mozilla/5.0 (compatible; Diffbot/1.0; +https://diffbot.com)'. 14% of top websites block Diffbot as of Aug 2026. No dedicated vendor robots.txt page was reachable (docs.diffbot.com crawler guide 301s to diffbot.com/docs/)."
    },
    {
      "id": "knownagents-directory",
      "title": "Known Agents — AI agent user-agent directory (formerly Dark Visitors)",
      "url": "https://knownagents.com/agents",
      "type": "dataset",
      "publisher": "Known Agents",
      "verified": "2026-08-20",
      "keyFindings": "darkvisitors.com/agents now 301s to knownagents.com/agents — the widely cited Dark Visitors directory has been rebranded, which matters for any audit that hard-codes darkvisitors URLs. Catalogue is typed (AI Agent, AI Assistant, AI Coding Agent, AI Data Provider, AI Data Scraper, AI Search Crawler, Undocumented AI Agent) and tracks per-agent robots.txt blocking rates across top websites. Newer agents catalogued include ChatGPT Agent, Google-Agent, GoogleAgent-Mariner, AmazonBuyForMe, Manus-User, NovaAct, Claude-Code, Gemini-Deep-Research, Google-NotebookLM, kagi-fetcher, Kimi-User, TongyiBot."
    },
    {
      "id": "knownagents-youbot",
      "title": "YouBot — Known Agents",
      "url": "https://knownagents.com/agents/youbot",
      "type": "dataset",
      "publisher": "Known Agents",
      "verified": "2026-08-20",
      "keyFindings": "Operated by You.com, typed AI Data Provider. 11% of top websites block YouBot as of Aug 2026; AI data providers as a class were 0.4% of all web traffic."
    },
    {
      "id": "langchain-html-splitters",
      "title": "Split HTML — LangChain text splitter integrations",
      "url": "https://docs.langchain.com/oss/python/integrations/splitters/split_html",
      "type": "vendor-doc",
      "publisher": "LangChain",
      "verified": "2026-08-20",
      "keyFindings": "Documented consumer of heading hierarchy in RAG pipelines. HTMLHeaderTextSplitter operates on <h1>,<h2>,<h3>… and 'adds metadata for each header \"relevant\" to any given chunk', preserving which headers preceded each chunk. Stated rationale: '(a) keeping related text grouped (more or less) semantically and (b) preserving context-rich information encoded in document structures.' HTMLSectionSplitter splits on sections via XSLT and — tellingly — falls back to inferring sections from FONT SIZE when header markup is absent, i.e. missing headings force a heuristic guess."
    },
    {
      "id": "lf-a2a-press-release",
      "title": "Linux Foundation Launches the Agent2Agent Protocol Project",
      "url": "https://www.linuxfoundation.org/press/linux-foundation-launches-the-agent2agent-protocol-project-to-enable-secure-intelligent-communication-between-ai-agents",
      "type": "article",
      "publisher": "The Linux Foundation",
      "verified": "2026-08-20",
      "keyFindings": "Dated 2025-06-23 (Open Source Summit NA). Google donated A2A; supporting founding partners quoted: AWS, Cisco (Outshift), Salesforce, SAP, Microsoft, ServiceNow. 'By joining the Linux Foundation, A2A is ensuring the long-term neutrality, collaboration and governance...' Notably does NOT mention Agent Cards or website-level discovery — framing is agent-to-agent, not site-to-agent."
    },
    {
      "id": "lh-a11ytree",
      "title": "Lighthouse core/audits/agentic/agent-accessibility-tree.js",
      "url": "https://raw.githubusercontent.com/GoogleChrome/lighthouse/main/core/audits/agentic/agent-accessibility-tree.js",
      "type": "repo",
      "publisher": "GoogleChrome/lighthouse",
      "verified": "2026-08-20",
      "keyFindings": "Pure re-slice of the existing axe-core Accessibility artifact: filters violations to a hardcoded 33-rule TARGET_RULES set (button-name, label, link-name, select-name, document-title, aria-*, tabindex, autocomplete-valid, svg-img-alt, ...). Binary score. No new gathering, no agent-specific semantics."
    },
    {
      "id": "lh-ard-pr",
      "title": "Lighthouse PR #17168 — new_audit(ard-schema): add Agent Resource Discovery gatherer and schema audit",
      "url": "https://github.com/GoogleChrome/lighthouse/pull/17168",
      "type": "repo",
      "publisher": "GoogleChrome/lighthouse",
      "verified": "2026-08-20",
      "keyFindings": "OPEN PR (created 2026-08-10, branch agentic-resource-discovery). Adds core/audits/agentic/ard-schema.js + core/gather/gatherers/agentic/ard.js + vendored third-party/ard/ard.js ConformanceTester. Discovery precedence implemented: robots.txt 'Agentmap:' > <link rel=\"ai-catalog\"> > Link: <...>; rel=ai-catalog HTTP header > /.well-known/ai-catalog.json fallback. Scores 1 / 0.5 (warnings) / 0 (errors); adds a Lighthouse-only warning for entries missing representativeQueries. This is the single biggest false-uniqueness risk for any ai-catalog.json check."
    },
    {
      "id": "lh-config",
      "title": "Lighthouse core/config/agentic-browsing-config.js (main branch)",
      "url": "https://raw.githubusercontent.com/GoogleChrome/lighthouse/main/core/config/agentic-browsing-config.js",
      "type": "repo",
      "publisher": "GoogleChrome/lighthouse",
      "verified": "2026-08-20",
      "keyFindings": "Complete shipped list of the Agentic Browsing category: exactly 6 auditRefs — agent-accessibility-tree, webmcp-form-coverage, webmcp-registered-tools, webmcp-schema-validity, cumulative-layout-shift, llms-txt. Two groups (webmcp, agent-accessibility). Category description says 'still under development and subject to change'. Copyright 2026 Google LLC."
    },
    {
      "id": "lh-gatherer-llmstxt",
      "title": "Lighthouse core/gather/gatherers/agentic/llms-txt.js",
      "url": "https://raw.githubusercontent.com/GoogleChrome/lighthouse/main/core/gather/gatherers/agentic/llms-txt.js",
      "type": "repo",
      "publisher": "GoogleChrome/lighthouse",
      "verified": "2026-08-20",
      "keyFindings": "Fetches only new URL('/llms.txt', finalDisplayedUrl) — hardcoded root path, single request, no llms-full.txt, no .md mirrors."
    },
    {
      "id": "lh-llmstxt",
      "title": "Lighthouse core/audits/agentic/llms-txt.js",
      "url": "https://raw.githubusercontent.com/GoogleChrome/lighthouse/main/core/audits/agentic/llms-txt.js",
      "type": "repo",
      "publisher": "GoogleChrome/lighthouse",
      "verified": "2026-08-20",
      "keyFindings": "Entire llms.txt audit logic is three regexes on the body of GET /llms.txt: /^\\s*#\\s+.+/m (H1), /\\[.+\\]\\(.+\\)/ (any markdown link), content.length < 50 (too short). 4xx => notApplicable, 5xx => score 0. No section parsing, no link resolution, no link liveness, no blockquote summary, no discovery via <link> or robots.txt."
    },
    {
      "id": "lh-webmcp",
      "title": "Lighthouse webmcp-registered-tools / webmcp-form-coverage / webmcp-schema-validity audits",
      "url": "https://raw.githubusercontent.com/GoogleChrome/lighthouse/main/core/audits/webmcp-form-coverage.js",
      "type": "repo",
      "publisher": "GoogleChrome/lighthouse",
      "verified": "2026-08-20",
      "keyFindings": "registered-tools and form-coverage are INFORMATIVE (never fail the score). All three return notApplicable when artifacts.WebMCP.isSupported is false — i.e. on any Chrome without the WebMCP flag the whole WebMCP group silently scores nothing. schema-validity only checks form-level toolname/tooldescription attributes and per-field name/description presence."
    },
    {
      "id": "lighthouse-a11y-scoring",
      "title": "Lighthouse accessibility scoring",
      "url": "https://developer.chrome.com/docs/lighthouse/accessibility/scoring",
      "type": "vendor-doc",
      "publisher": "Google / Chrome for Developers",
      "verified": "2026-08-20",
      "keyFindings": "Accessibility score is a weighted average of binary pass/fail audits. Critically, the weights are not invented: 'Weighting is based on axe user impact assessments', linking out to the axe-core rule-descriptions doc. Weight tiers 10 / 7 / 3 map to axe impact levels. Explicit statement: 'Unlike the Performance audits, a page doesn't get points for partially passing an accessibility audit.' This is the single best precedent for sourcing audit weights to an external published authority rather than editorial judgement."
    },
    {
      "id": "lighthouse-architecture-doc",
      "title": "Lighthouse Architecture (docs/architecture.md)",
      "url": "https://github.com/GoogleChrome/lighthouse/blob/main/docs/architecture.md",
      "type": "vendor-doc",
      "publisher": "GoogleChrome/lighthouse",
      "verified": "2026-08-20",
      "keyFindings": "Defines the Gatherer -> Artifact -> Audit -> Category pipeline. 'The audits are tests for a single feature/optimization/metric. Using the Artifacts as input, an audit evaluates a test and resolves to a numeric score.' Category = 'roll-up collection of audits and audit groups into a user-facing section of the report'. Each audit ships a success title, failure title and a description explaining why it matters — the per-audit documentation contract."
    },
    {
      "id": "lighthouse-llms-txt-audit-source",
      "title": "Lighthouse core/audits/agentic/llms-txt.js (source code)",
      "url": "https://github.com/GoogleChrome/lighthouse/blob/main/core/audits/agentic/llms-txt.js",
      "type": "vendor-doc",
      "publisher": "Google Chrome / Lighthouse",
      "verified": "2026-08-21",
      "keyFindings": "PRIMARY SOURCE, read via GitHub API. Copyright 2026 Google LLC. Audit id 'llms-txt'. Exact logic: hasH1 = /^\\s*#\\s+.+/m.test(content); hasLink = /\\[.+\\]\\(.+\\)/.test(content); isTooShort = content.length < 50. Score = 1 only if all three pass. HTTP >=500 => score 0; HTTP 4xx (incl. 404) => notApplicable with score 1, i.e. ABSENCE IS NOT PENALIZED. Failure strings: 'File is missing a required H1 header (e.g., \"# Title\").', 'File does not appear to contain any links.', 'File is suspiciously short.' Description asserts LLMs 'may not be able to understand how you want your website to be crawled or used for training' — an unsourced claim by Google. Note: no blockquote check, no link-validity check, no description check."
    },
    {
      "id": "lighthouse-llms-txt-gatherer-source",
      "title": "Lighthouse core/gather/gatherers/agentic/llms-txt.js (source code)",
      "url": "https://github.com/GoogleChrome/lighthouse/blob/main/core/gather/gatherers/agentic/llms-txt.js",
      "type": "vendor-doc",
      "publisher": "Google Chrome / Lighthouse",
      "verified": "2026-08-20",
      "keyFindings": "PRIMARY SOURCE. Gatherer resolves exactly `new URL('/llms.txt', finalDisplayedUrl)` and fetches it. Decisive counter-evidence for the link-tag signal: Lighthouse does NOT read <link rel=\"describedby\">, does NOT check /.well-known/llms.txt, and does NOT fetch llms-full.txt. Root path only."
    },
    {
      "id": "lighthouse-new-audits-doc",
      "title": "Lighthouse: proposing a new audit (docs/new-audits.md)",
      "url": "https://github.com/GoogleChrome/lighthouse/blob/main/docs/new-audits.md",
      "type": "vendor-doc",
      "publisher": "GoogleChrome/lighthouse",
      "verified": "2026-08-20",
      "keyFindings": "Published admission criteria for a new audit: must be 'applicable to a significant portion of web developers (based on scale and severity of impact)', 'contribute significantly towards making the mobile web experience better for end users', 'be new, not something already measured', 'be measurable ... or have clear pass/fail states', 'be actionable', and 'not use 3rd party APIs for completing the audit check'. Also standardises detail types (node, source-location, url, bytes...) and audit-ID naming policy (no verbs, no no- prefixes). This is the governance artifact no AI-readiness tool currently has."
    },
    {
      "id": "lighthouse-perf-scoring",
      "title": "Lighthouse performance scoring",
      "url": "https://developer.chrome.com/docs/lighthouse/performance/performance-scoring",
      "type": "vendor-doc",
      "publisher": "Google / Chrome for Developers",
      "verified": "2026-08-20",
      "keyFindings": "Documents exact metric weights per Lighthouse major version (LH10: FCP 10%, SI 10%, LCP 25%, TBT 30%, CLS 25%) and explicitly contrasts them with LH8 weights (TTI 10%, CLS 15%), i.e. weight changes are versioned and published. Scoring curve is a log-normal derived from HTTP Archive field data: 25th percentile = score 50, 8th percentile = score 90. Includes a dedicated section on score variability and its external causes."
    },
    {
      "id": "lighthouse-preconnect",
      "title": "Preconnect to required origins (uses-rel-preconnect)",
      "url": "https://developer.chrome.com/docs/lighthouse/performance/uses-rel-preconnect",
      "publisher": "Google Chrome Developers",
      "type": "vendor-doc",
      "keyFindings": "\"This audit has moved into the Network dependency tree insight as of Lighthouse 13.\" Warns against overuse: an unused preconnect is \"especially bad if the connection isn't used within 10 seconds, as the browser closes it, wasting all of that early connection work\", and preconnect \"can still take up valuable CPU time, particularly on secure connections\". Lighthouse flags specific beneficial origins rather than testing for mere presence.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "lighthouse-scorecalc",
      "title": "Lighthouse Scoring Calculator",
      "url": "https://googlechrome.github.io/lighthouse/scorecalc/",
      "type": "vendor-doc",
      "publisher": "Google Chrome",
      "verified": "2026-08-20",
      "keyFindings": "Public interactive calculator letting anyone reproduce a Lighthouse score from raw metric values, including the log-normal curve behaviour. Demonstrates the 'anyone can re-derive our number' standard of transparency. Notes the calculator does not model scores below 5/100 because of log-normal tail behaviour."
    },
    {
      "id": "lighthouse-scoring-doc",
      "title": "Lighthouse Scores (docs/scoring.md)",
      "url": "https://github.com/GoogleChrome/lighthouse/blob/main/docs/scoring.md",
      "type": "vendor-doc",
      "publisher": "GoogleChrome/lighthouse (Apache-2.0, ~30.7k stars)",
      "verified": "2026-08-20",
      "keyFindings": "Publishes the full per-audit weight table (e.g. every a11y audit at 4.1%), states Best Practices audits are equally weighted (~6 pts each) and SEO audits equally weighted (~8 pts each) with Structured Data explicitly unscored/manual. Links to the exact line in core/config/default-config.js as machine-readable source of truth, and links back to the v6.5.0 version of the same doc so historical weightings remain inspectable. A comment documents the regeneration script (core/scripts/print-a11y-scoring.js) — weights are generated, not hand-maintained."
    },
    {
      "id": "llms-txt-core-source",
      "title": "AnswerDotAI/llms-txt — core.py",
      "url": "https://raw.githubusercontent.com/AnswerDotAI/llms-txt/main/llms_txt/core.py",
      "type": "repo",
      "publisher": "Answer.AI",
      "verified": "2026-08-21",
      "keyFindings": "The reference parser emits each link as an object with `title`, `url` and an optional `desc`, making the note a first-class parsed field rather than free text."
    },
    {
      "id": "llms-txt-pypi",
      "title": "llms-txt on PyPI",
      "url": "https://pypi.org/project/llms-txt",
      "type": "repo",
      "publisher": "Answer.AI",
      "verified": "2026-08-21",
      "keyFindings": "The `llms_txt2ctx` CLI expands the links listed in an llms.txt into a single LLM context document — the concrete consumption path for link descriptions."
    },
    {
      "id": "llmstxt-directory-cloud",
      "title": "llms.txt directory",
      "url": "https://directory.llmstxt.cloud/",
      "type": "dataset",
      "publisher": "llmstxt.cloud",
      "verified": "2026-08-20",
      "keyFindings": "Curated directory of sites adopting llms.txt: 849 websites across 21 pages of listings, categorized as Products 447, Developer tools 358, AI 187, Finance 167. Notable adopters with token counts: Anthropic Claude (892 tokens), Perplexity (4K), Cloudflare (49K), Vercel AI SDK (293K), Hugging Face, Zapier (14K), Coinbase (74K). Curated/opt-in, so it measures visible developer-tool adoption, not web-wide adoption."
    },
    {
      "id": "llmstxt-github-repo",
      "title": "AnswerDotAI/llms-txt repository",
      "url": "https://github.com/AnswerDotAI/llms-txt",
      "type": "draft-spec",
      "publisher": "Answer.AI",
      "verified": "2026-08-20",
      "keyFindings": "Spec home. ~2.6k stars, 159 forks, 122 commits, last modified 2026-08-10. Does not name any AI system that consumes llms.txt; only notes AI labs publish their own. No conformance suite, no governance body, no version registry."
    },
    {
      "id": "llmstxt-org-changes",
      "title": "llms.txt v2 changes page",
      "url": "https://llmstxt.org/changes.html",
      "type": "draft-spec",
      "publisher": "Jeremy Howard / Answer.AI",
      "verified": "2026-08-20",
      "keyFindings": "v2 changelog: adds rel=alternate/describedby link relations; relaxes .md URL pattern (v1 required page.html.md, v2 permits page.md); adds hierarchical/path-scoped llms.txt with most-specific-wins; REMOVES the llms_txt2ctx tool and the mechanical meaning of 'Optional' sections. Author reframes v2 around 'how agents actually use websites' rather than predicting agent behavior. Still no llms-full.txt and no .well-known guidance."
    },
    {
      "id": "llmstxt-org-v2",
      "title": "The /llms.txt file, v2",
      "url": "https://llmstxt.org/",
      "type": "draft-spec",
      "publisher": "Jeremy Howard / Answer.AI",
      "verified": "2026-08-21",
      "keyFindings": "Canonical proposal. Published 2024-09-03, v2 dated 2026-08-10. Format: optional BOM, H1 with project name ('the only required section'), blockquote summary, zero+ markdown sections, zero+ H2 'file list' sections of [name](url) links with optional notes. v2 ADDS link relations: rel=\"alternate\" type=\"text/markdown\" -> markdown page version, rel=\"describedby\" -> the covering llms.txt, deliverable as HTML <link> or HTTP Link: header. Two .md URL patterns allowed (page.html.md or page.md). NO mention of llms-full.txt anywhere in v1 or v2. Claims 'the AI labs themselves publish llms.txt files' (OpenAI, Anthropic, Gemini) — a publisher-side, not consumer-side, claim."
    },
    {
      "id": "llmstxt-repo",
      "title": "AnswerDotAI/llms-txt",
      "url": "https://github.com/AnswerDotAI/llms-txt",
      "publisher": "Answer.AI",
      "type": "repo",
      "keyFindings": "README (13,622 bytes, fetched via GitHub API) contains no mention of llms-full.txt. Issue #55 'llms.full.txt generated is too large' is open with no maintainer response on the file's status.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "llmstxt-site",
      "title": "llmstxt.site — indexed llms.txt files",
      "url": "https://llmstxt.site/",
      "type": "dataset",
      "publisher": "llmstxt.site",
      "verified": "2026-08-20",
      "keyFindings": "Searchable index of llms.txt files derived from the llmstxt.org standard; roughly 600+ indexed sites. Tracks per-entry: product name, website URL, llms.txt location, llms.txt token count, llms-full.txt location and token count (0 to over 1.5M tokens). Useful for file-size/complexity distribution; second independent directory to cross-check llmstxt.cloud's 849."
    },
    {
      "id": "llmstxt-spec",
      "title": "The /llms.txt file",
      "url": "https://llmstxt.org/",
      "type": "draft-spec",
      "publisher": "Answer.AI (Jeremy Howard)",
      "verified": "2026-08-21",
      "keyFindings": "Defines llms.txt at /llms.txt (or any subpath) in Markdown, with H1 title, optional blockquote summary and H2-delimited link sections. Recommends clean Markdown mirrors 'at the same URL as the original page, either with .md appended (page.html.md) or with the extension replaced by .md (page.md)', and uses type=\"text/markdown\" in link relations. CRITICALLY: the spec states NO requirement for the file's own HTTP Content-Type, no CORS guidance and no caching guidance — so any content-type audit is enforcing convention, not spec."
    },
    {
      "id": "llmstxt-spec-link",
      "title": "The /llms.txt file — link relation guidance",
      "url": "https://llmstxt.org/",
      "publisher": "Answer.AI (Jeremy Howard)",
      "type": "spec",
      "keyFindings": "Only link relation recommended is '</docs/page.html.md>; rel=\"alternate\"; type=\"text/markdown\"' for per-page markdown. No text/plain relation, no llms-full.txt, no head-link discovery for any root file.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "llmstxt-spec-raw",
      "title": "llmstxt.org raw specification source (index.md)",
      "url": "https://llmstxt.org/index.md",
      "publisher": "Answer.AI",
      "type": "spec",
      "keyFindings": "Raw spec text confirmed to contain zero occurrences of the string 'llms-full.txt'.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "llmtxt-info-observatory",
      "title": "llmtxt.info — llms.txt reference and adoption Observatory",
      "url": "https://llmtxt.info/",
      "type": "dataset",
      "publisher": "llmtxt.info (powered by abovefear/llms-txt-toolkit)",
      "verified": "2026-08-20",
      "keyFindings": "Publishes an adoption Observatory with dated measurements — 117 of 218 fixed-panel hosts served a file as of 2026-08-17, 303 linked pages checked in the latest discovery snapshot — plus 'live verification receipts using a versioned validator', a dedicated /methodology/ page, and 'claims you can inspect' linking back to the raw observatory. The only entity in this landscape already doing versioned, reproducible, publicly inspectable measurement. A citable adoption source for us and a template for how our evidence pages should link claim -> receipt."
    },
    {
      "id": "longato-md-files-study",
      "title": "Do LLMs Crawl Markdown (.md) Files? Data Analysis",
      "url": "https://www.longato.ch/llm-md-files/",
      "type": "study",
      "publisher": "Flavio Longato",
      "verified": "2026-08-20",
      "keyFindings": "24 hours of CDN logs across two DA>90 sites with millions of daily visits, monitoring GPTBot, ClaudeBot, PerplexityBot. Zero requests from recognized LLM bots to .md files that were explicitly listed in llms.txt; only random scraping bots. Author states the 24h window is the honest limitation (a full week exceeded 150GB). Does not test Accept headers/content negotiation — .md URL discovery only."
    },
    {
      "id": "lost-in-middle",
      "title": "Lost in the Middle: How Language Models Use Long Contexts",
      "url": "https://arxiv.org/abs/2307.03172",
      "type": "study",
      "publisher": "Liu et al. (arXiv 2307.03172)",
      "verified": "2026-08-20",
      "keyFindings": "\"performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts\", including for explicitly long-context models. Grounds position-sensitive checks: burying the answer behind kilotokens of preamble is not merely a cost problem, it measurably lowers retrieval accuracy."
    },
    {
      "id": "lost-in-the-middle-arxiv",
      "title": "Lost in the Middle: How Language Models Use Long Contexts",
      "url": "https://arxiv.org/abs/2307.03172",
      "type": "study",
      "publisher": "Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni, Liang (arXiv / TACL)",
      "verified": "2026-08-20",
      "keyFindings": "Mechanistic support for answer-first placement, at the context level: 'performance can degrade significantly when changing the position of relevant information'; 'performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models.' Important scope caveat: this is position within the model's input context (i.e. which retrieved chunk sits where), not position within a publisher's HTML page."
    },
    {
      "id": "machine-readable-ads-paper",
      "title": "Machine-Readable Ads: Accessibility and Trust Patterns for AI Web Agents interacting with Online Advertisements",
      "url": "https://arxiv.org/abs/2507.12844",
      "type": "study",
      "publisher": "arXiv (Nitu, Mühle, Stöckl, 2025)",
      "verified": "2026-08-20",
      "keyFindings": "Controlled 300+ trial experiment across GPT-4o, Claude 3.7 Sonnet, Gemini 2.0 Flash and OpenAI Operator on a news-site clone. Headline finding directly supports the accessible-name thesis: agents 'never scroll beyond two viewports and ignore purely visual calls to action, clicking banners only when semantic button overlays or off-screen text labels are present.' Derived design principles include semantic overlays and hidden text labels. Caveat: the domain is advertisements, and the manipulation is presence/absence of a text label rather than a graded semantic-HTML quality scale."
    },
    {
      "id": "maxaeo-citation-study",
      "title": "Does llms.txt Work? Evidence From AI Citation Data",
      "url": "https://maxaeo.ai/blog/does-llms-txt-work/",
      "type": "study",
      "publisher": "MaxAEO",
      "verified": "2026-08-20",
      "keyFindings": "Largest outcome study found. 2,400 domains (326 with llms.txt), Feb 1 – Apr 30 2026, citations tracked across 8 AI surfaces (ChatGPT, Gemini, Perplexity, Claude, Copilot, Grok, Google AI Mode, AI Overviews), 240 matched pairs controlling for category, organic visibility and content volume. RESULT: citation rate 11.8% with llms.txt vs 11.6% without = +0.2pp, inside the ±0.5pp noise band. Before/after on 58 deploying sites: median +0.1pp vs +0.2pp for controls. Crawler side: 19 customer sites saw 41 total /llms.txt requests against ~1.1M AI-crawler page fetches."
    },
    {
      "id": "mcp-code-search-link-rel",
      "title": "GitHub code search: rel=\"mcp\" in the MCP specification repository",
      "url": "https://github.com/modelcontextprotocol/modelcontextprotocol",
      "publisher": "GitHub (code search API)",
      "type": "repo",
      "keyFindings": "Search for rel=\"mcp\" scoped to repo:modelcontextprotocol/modelcontextprotocol returns total_count=0 — the HTML link-rel discovery mechanism the audit checks for does not exist in any MCP spec, SEP, draft, or blog post. By contrast '.well-known/mcp.json' does appear (seps/2575-stateless-mcp.md, docs/seps/2575-stateless-mcp.mdx, blog/content/posts/2025-12-19-mcp-transport-future.md), confirming the well-known path is the discussed mechanism.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "mcp-ext-server-card-discovery",
      "title": "experimental-ext-server-card — docs/discovery.md",
      "url": "https://raw.githubusercontent.com/modelcontextprotocol/experimental-ext-server-card/main/docs/discovery.md",
      "type": "draft-spec",
      "publisher": "Model Context Protocol",
      "verified": "2026-08-20",
      "keyFindings": "'Protocol Revision: draft'. 'An AI Catalog MAY be served from any URL. For automated domain-level discovery, hosts MAY publish one at: /.well-known/ai-catalog.json. Clients performing domain-level discovery SHOULD attempt to retrieve this well-known URL. When served over HTTP, the document SHOULD use the `application/ai-catalog+json` media type.' Entry type for MCP is `application/mcp-server-card+json`; identifiers use `urn:air:{publisher}:{namespace}:{name}`. Repo has only 5 stars (created 2026-04-27, last push 2026-08-12)."
    },
    {
      "id": "mcp-http",
      "title": "MCP specification — Streamable HTTP transport (draft, rev 2026-07-28)",
      "url": "https://modelcontextprotocol.io/specification/draft/basic/transports/streamable-http",
      "type": "spec",
      "publisher": "Model Context Protocol",
      "verified": "2026-08-20",
      "keyFindings": "Revision 2026-07-28 REMOVED protocol-level sessions (Mcp-Session-Id), the GET stream endpoint, and Last-Event-ID resumption. Single POST-only MCP endpoint. Every POST MUST carry MCP-Protocol-Version, Mcp-Method, and Mcp-Name (for tools/call, resources/read, prompts/get); Accept MUST list both application/json and text/event-stream. Header/body mismatch MUST be rejected with 400 + JSON-RPC -32020 HeaderMismatch. Unknown method MUST be 404 + -32601. Origin MUST be validated (403 on invalid). Legacy servers are detected by falling back to initialize only when the 400/404/405 body is NOT a recognised modern JSON-RPC error."
    },
    {
      "id": "mcp-instructions-field",
      "title": "MCP Specification 2026-07-28 — Discovery (DiscoverResult.instructions)",
      "url": "https://modelcontextprotocol.io/specification/2026-07-28/server/discover.md",
      "publisher": "Model Context Protocol",
      "type": "spec",
      "keyFindings": "Shows where natural-language model guidance actually lives in a live standard: DiscoverResult carries an `instructions` field, 'Optional natural-language guidance for LLMs on how to use this server effectively.' Demonstrates the audit's underlying intent is legitimate but is served by a real field in a real protocol, not by an unregistered OpenAPI extension.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "mcp-official-registry-api",
      "title": "Official MCP Registry API (v0/servers)",
      "url": "https://registry.modelcontextprotocol.io/v0/servers",
      "type": "spec",
      "publisher": "Model Context Protocol project",
      "verified": "2026-08-20",
      "keyFindings": "Live and publicly queryable without auth; returns server records with $schema (2025-12-11 server schema), name, description, version, remotes, and _meta registry status/publishedAt/updatedAt fields. Cursor-paginated; metadata returns only the page 'count', NOT a global total — there is no official published server count. Records observed with 2026 publish dates, confirming the registry is actively receiving submissions."
    },
    {
      "id": "mcp-pr-2127-open",
      "title": "SEP-2127: MCP Server Cards - HTTP Server Discovery (open PR)",
      "url": "https://github.com/modelcontextprotocol/modelcontextprotocol/pull/2127",
      "publisher": "Model Context Protocol",
      "type": "spec",
      "keyFindings": "State: OPEN (created 2026-01-21, not merged as of this research). Successor to SEP-1649. Now proposes '.well-known/ai-catalog.json: HTTP endpoint for pre-connection discovery' — renamed away from mcp.json. Domain-level MCP discovery therefore remains an unratified draft, and the endpoint name has already changed, so no audit should be scored against it yet.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "mcp-registry-github",
      "title": "modelcontextprotocol/registry — official registry source",
      "url": "https://github.com/modelcontextprotocol/registry",
      "type": "spec",
      "publisher": "Model Context Protocol project",
      "verified": "2026-08-20",
      "keyFindings": "Community-driven registry service for MCP servers, functioning as an app-store-style catalog for MCP clients. Status is explicitly PREVIEW, not GA: 'breaking changes or data resets may occur.' API freeze at v0.1 declared 2025-10-24. README publishes no server counts or adoption metrics."
    },
    {
      "id": "mcp-registry-repo",
      "title": "modelcontextprotocol/registry — official community MCP server registry",
      "url": "https://github.com/modelcontextprotocol/registry",
      "type": "vendor-doc",
      "publisher": "Model Context Protocol",
      "verified": "2026-08-20",
      "keyFindings": "7,180 stars, created 2025-02-05, last push 2026-08-19. Registry entries use `server.json`. SEP-2127 states the Registry's server.json 'separately describes registry entries' and is a distinct schema from Server Cards — publication to the registry is a submission workflow, not website-derived discovery."
    },
    {
      "id": "mcp-sep-1649",
      "title": "SEP-1649: MCP Server Cards — HTTP Server Discovery via .well-known (superseded issue)",
      "url": "https://github.com/modelcontextprotocol/modelcontextprotocol/issues/1649",
      "type": "draft-spec",
      "publisher": "Model Context Protocol",
      "verified": "2026-08-20",
      "keyFindings": "Opened 2025-10-14, 49 comments, label 'draft', CLOSED 2026-01-26 (superseded by SEP-2127). Original proposal used `/.well-known/mcp/server-card.json` — a path that some vendors shipped and that the current SEP no longer specifies."
    },
    {
      "id": "mcp-sep-1649-server-cards",
      "title": "SEP-1649: MCP Server Cards — HTTP Server Discovery via .well-known",
      "url": "https://github.com/modelcontextprotocol/modelcontextprotocol/issues/1649",
      "publisher": "Model Context Protocol (authors @dsp-ant, @nickcoai)",
      "type": "spec",
      "keyFindings": "Status: Draft. Created 2025-10-14, closed COMPLETED 2026-01-26 solely to migrate to a PR-based workflow ('This SEP has been moved to a PR-based workflow per SEP-1850. Continued in: #2127'). Proposes .well-known/mcp.json to fix stated pain points including 'No Domain-Level Discovery: Clients cannot automatically discover available MCP servers on a domain.' Confirms the audit's underlying need is real and Anthropic-sponsored — and that the sanctioned mechanism is a well-known JSON endpoint, never an HTML link tag.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "mcp-sep-2127",
      "title": "SEP-2127: MCP Server Cards — HTTP Server Discovery (pull request)",
      "url": "https://github.com/modelcontextprotocol/modelcontextprotocol/pull/2127",
      "type": "draft-spec",
      "publisher": "Model Context Protocol (dsp-ant)",
      "verified": "2026-08-20",
      "keyFindings": "Opened 2026-01-21, labels ['SEP','in-review','extension','roadmap/transport'], state OPEN, NOT merged as of 2026-08-11. Current text: 'Cards themselves can be hosted at any unreserved URI, with `<streamable-http-url>/server-card` reserved as the recommended location.' Domain-level discovery is delegated to an AI Catalog at `/.well-known/ai-catalog.json`. Explicitly names the pain point: 'No Domain-Level Discovery: Clients cannot automatically discover available MCP servers on a domain.' Deliberately EXCLUDES tools/resources/prompts from the card."
    },
    {
      "id": "mcp-server-tools",
      "title": "Model Context Protocol — server tools",
      "url": "https://modelcontextprotocol.io/specification/2025-06-18/server/tools",
      "type": "spec",
      "publisher": "Model Context Protocol",
      "verified": "2026-08-21",
      "keyFindings": "`description` is the \"Human-readable description of functionality\"; per-property descriptions live in the `inputSchema`."
    },
    {
      "id": "mcp-spec-2025-06-18-transports",
      "title": "MCP Specification 2025-06-18 — Transports",
      "url": "https://modelcontextprotocol.io/specification/2025-06-18/basic/transports",
      "type": "spec",
      "publisher": "Model Context Protocol / Anthropic",
      "verified": "2026-08-20",
      "keyFindings": "Defines only stdio and Streamable HTTP. States 'The server MUST provide a single HTTP endpoint path (hereafter referred to as the MCP endpoint)... For example, this could be a URL like https://example.com/mcp.' Contains ZERO .well-known discovery mechanism. Backwards-compatibility section explicitly says clients 'Accept an MCP server URL from the user' — i.e. manual configuration is the specified discovery path."
    },
    {
      "id": "mcp-spec-2026-discovery",
      "title": "MCP Specification 2026-07-28 — Discovery (server/discover)",
      "url": "https://modelcontextprotocol.io/specification/2026-07-28/server/discover.md",
      "publisher": "Model Context Protocol",
      "type": "spec",
      "keyFindings": "The current ratified discovery mechanism is a JSON-RPC method, not web discovery: 'server/discover lets a client query a server's supported protocol versions, capabilities, and identity before sending any other requests. Servers MUST implement it.' It is sent to an endpoint the client already has, so it cannot help an agent find a server from a website. No /.well-known/ path and no HTML link tag appear anywhere on the page.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "mcp-spec-authorization",
      "title": "Model Context Protocol Specification (2025-11-25) — Authorization",
      "url": "https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization",
      "type": "spec",
      "publisher": "Model Context Protocol",
      "verified": "2026-08-20",
      "keyFindings": "Under 'Communication Security': implementations MUST follow OAuth 2.1 §1.5, and specifically '1. All authorization server endpoints MUST be served over HTTPS. 2. All redirect URIs MUST be either localhost or use HTTPS.' Client ID Metadata Documents MUST be hosted at an HTTPS URL. Authorization servers 'SHOULD cache metadata respecting HTTP cache headers' — one of the few places an agent spec explicitly defers to HTTP caching."
    },
    {
      "id": "mcp-spec-authorization-rfc9728",
      "title": "MCP Specification 2026-07-28 — Authorization (RFC 9728 protected-resource metadata)",
      "url": "https://raw.githubusercontent.com/modelcontextprotocol/modelcontextprotocol/main/docs/specification/2026-07-28/basic/authorization/index.mdx",
      "type": "spec",
      "publisher": "Model Context Protocol",
      "verified": "2026-08-20",
      "keyFindings": "'MCP servers MUST implement OAuth 2.0 Protected Resource Metadata (RFC9728)'. Example: `WWW-Authenticate: Bearer resource_metadata=\"https://mcp.example.com/.well-known/oauth-protected-resource\"`. This is the ONLY .well-known URI normatively required by MCP — and it lives on the MCP server's origin, describing auth, not on the website, and it does not let anyone find the server."
    },
    {
      "id": "mcp-spec-server-discover-rpc",
      "title": "MCP Specification 2026-07-28 — server/discover",
      "url": "https://raw.githubusercontent.com/modelcontextprotocol/modelcontextprotocol/main/docs/specification/2026-07-28/server/discover.mdx",
      "type": "spec",
      "publisher": "Model Context Protocol",
      "verified": "2026-08-20",
      "keyFindings": "'`server/discover` lets a client query a server's supported protocol versions, capabilities, and identity before sending any other requests. Servers MUST implement it.' This is an in-protocol JSON-RPC method against a known endpoint — it is NOT website-level discovery."
    },
    {
      "id": "mcp-spec-versioning",
      "title": "Model Context Protocol — Versioning (current revision 2026-07-28)",
      "url": "https://modelcontextprotocol.io/specification/versioning",
      "type": "spec",
      "publisher": "Model Context Protocol / Anthropic",
      "verified": "2026-08-20",
      "keyFindings": "Current protocol revision is 2026-07-28. Revisions listed: 2024-11-05, 2025-03-26, 2025-06-18, 2025-11-25, 2026-07-28, draft. Version negotiation happens in-band via `io.modelcontextprotocol/protocolVersion` and the `MCP-Protocol-Version` header, plus a mandatory `server/discover` RPC — all of which require the client to ALREADY hold the server URL."
    },
    {
      "id": "mcpdoc",
      "title": "langchain-ai/mcpdoc — MCP server for llms.txt",
      "url": "https://raw.githubusercontent.com/langchain-ai/mcpdoc/main/README.md",
      "publisher": "LangChain",
      "type": "repo",
      "keyFindings": "Closest thing to a documented consumer: a fetch_docs tool that reads URLs inside provided llms.txt files, configured for Cursor, Windsurf and Claude Code. Reads llms.txt only; never mentions llms-full.txt.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "mdn-cors",
      "title": "Cross-Origin Resource Sharing (CORS)",
      "url": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/CORS",
      "type": "article",
      "publisher": "MDN Web Docs (Mozilla)",
      "verified": "2026-08-20",
      "keyFindings": "Decisive for the CORS signal: 'For security reasons, browsers restrict cross-origin HTTP requests initiated from scripts.' CORS is enforced by the user agent, not the server; the server merely opts in. Non-browser HTTP clients — which is what GPTBot, ClaudeBot, PerplexityBot and server-side agent backends are — do not implement the same-origin policy and are therefore unaffected by missing Access-Control-Allow-Origin."
    },
    {
      "id": "mdn-csp",
      "title": "Content Security Policy guide",
      "url": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/CSP",
      "type": "vendor-doc",
      "publisher": "MDN / Mozilla",
      "verified": "2026-08-20",
      "keyFindings": "script-src constrains which JS may load and execute (blocking external attacker-controlled scripts, inline scripts, inline handlers, javascript: URLs, eval); strict CSP with nonces/hashes is recommended over allowlists. frame-ancestors addresses framing. Gives a machine-checkable definition of 'this page's DOM writers are constrained'."
    },
    {
      "id": "mdn-getelementbyid",
      "title": "MDN — Document.getElementById()",
      "url": "https://developer.mozilla.org/en-US/docs/Web/API/Document/getElementById",
      "type": "vendor-doc",
      "publisher": "MDN",
      "verified": "2026-08-21",
      "keyFindings": "\"IDs should be unique inside a document. If two or more elements in a document have the same ID, this method returns the first element found.\""
    },
    {
      "id": "mdn-iframe",
      "title": "MDN — the iframe element",
      "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/iframe",
      "type": "vendor-doc",
      "publisher": "MDN",
      "verified": "2026-08-21",
      "keyFindings": "\"People navigating with assistive technology such as a screen reader can use the `title` attribute on an `<iframe>` to label its content\"; without it they must navigate into the frame to determine its contents."
    },
    {
      "id": "mdn-meta-http-equiv",
      "title": "MDN — meta http-equiv",
      "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/meta/http-equiv",
      "type": "vendor-doc",
      "publisher": "MDN",
      "verified": "2026-08-21",
      "keyFindings": "With a non-negative integer the page \"reloads after that many seconds\"; followed by `;url=` it \"redirects to that URL after the specified delay\". The timer starts after `load`/`pageshow`."
    },
    {
      "id": "mdn-permissions-policy",
      "title": "Permissions-Policy header",
      "url": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Permissions-Policy",
      "publisher": "MDN Web Docs / Mozilla",
      "type": "vendor-doc",
      "keyFindings": "\"Directives have a default allowlist, which is always one of *, self, or none for the Permissions-Policy HTTP header, and governs the default behavior if they are not explicitly listed in a policy.\" Prompts occur only when the page's own code requests the feature and permission is undecided. Absence of the header does not itself cause prompts. Header primarily constrains embedded iframes.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "mdn-sri",
      "title": "Subresource Integrity",
      "url": "https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity",
      "type": "vendor-doc",
      "publisher": "MDN / Mozilla",
      "verified": "2026-08-20",
      "keyFindings": "integrity= pins a cryptographic hash on <script> and <link>; the browser refuses the resource on mismatch. Explicit threat model: 'if an attacker gains control of the third-party host, then they can inject arbitrary malicious content into its files.' Quantifies the unpinned third-party surface that can write into the DOM an agent later reads."
    },
    {
      "id": "mdn-title-element",
      "title": "MDN — the title element",
      "url": "https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/title",
      "type": "vendor-doc",
      "publisher": "MDN",
      "verified": "2026-08-21",
      "keyFindings": "The title \"defines the document's title that is shown in a browser's title bar or a page's tab\"; reading the page title is \"a common navigation technique for users of assistive technology\"."
    },
    {
      "id": "meta-crawlers",
      "title": "Meta web crawlers",
      "url": "https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/",
      "publisher": "Meta",
      "type": "vendor-doc",
      "keyFindings": "meta-externalagent 'crawls the web for use cases such as training foundation AI models'; documented robots.txt sample shows 'Allow: / # Allow everything' and 'Disallow: /private/ # Disallow a specific directory'. Notes robots.txt may be cached up to 24 hours.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "meta-sharing-webmasters",
      "title": "Meta — sharing webmaster guide",
      "url": "https://developers.facebook.com/docs/sharing/webmasters",
      "type": "vendor-doc",
      "publisher": "Meta",
      "verified": "2026-08-21",
      "keyFindings": "Instructs sites to add `og:url` (\"The canonical URL for your page\"), `og:title`, `og:description` and `og:image`, and names the crawler user-agent `facebookexternalhit/1.1`."
    },
    {
      "id": "meta-web-crawlers-docs",
      "title": "Meta Web Crawlers",
      "url": "https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/",
      "type": "vendor-doc",
      "publisher": "Meta",
      "verified": "2026-08-21",
      "keyFindings": "meta-externalagent: 'crawls the web for use cases such as training foundation AI models or improving products by indexing content directly' — respects robots.txt. meta-externalfetcher: 'fetches individual links at a user's request' and 'may bypass robots.txt rules'. Meta-WebIndexer (new): 'navigates the web to improve Meta AI search result quality for users' and helps 'cite and link to your content in Meta AI's responses' — respects robots.txt. Meta-ExternalAds: ad/business crawling. facebookexternalhit: may bypass robots.txt for 'security or integrity checks, such as checking for malware or malicious content'."
    },
    {
      "id": "microsoft-ads-ai-search-optimization",
      "title": "Optimizing Your Content for Inclusion in AI Search Answers",
      "url": "https://about.ads.microsoft.com/en/blog/post/october-2025/optimizing-your-content-for-inclusion-in-ai-search-answers",
      "type": "vendor-doc",
      "publisher": "Microsoft Advertising",
      "verified": "2026-08-21",
      "keyFindings": "\"Schema is a type of code that helps search engines and AI systems understand your content\", usually added in JSON-LD. Names four types: product, review, FAQ, event. Notably hedged — it says schema turns \"plain text into structured data that machines can interpret with confidence\" but never claims a Microsoft AI system parses the markup as a ranking or citation input. Weakest form of vendor endorsement: recommendation without a stated consumption mechanism."
    },
    {
      "id": "microsoft-nlweb-announcement",
      "title": "Introducing NLWeb: Bringing conversational interfaces directly to the web",
      "url": "https://news.microsoft.com/source/features/company-news/introducing-nlweb-bringing-conversational-interfaces-directly-to-the-web/",
      "type": "vendor-doc",
      "publisher": "Microsoft",
      "verified": "2026-08-20",
      "keyFindings": "Dated 2025-05-19 (Build 2025). 'Every NLWeb instance is also a Model Context Protocol (MCP) server, allowing websites to make their content discoverable and accessible to agents.' 'NLWeb leverages semi-structured formats like Schema.org, RSS and other data that websites already publish.' Named early adopters: Chicago Public Media, Common Sense Media, DDM (Allrecipes/Serious Eats), Eventbrite, Hearst (Delish), Inception Labs, Milvus, O'Reilly Media, Qdrant, Shopify, Snowflake, Tripadvisor."
    },
    {
      "id": "mintlify-llmstxt",
      "title": "llms.txt — Mintlify docs",
      "url": "https://mintlify.com/docs/ai/llmstxt",
      "type": "vendor-doc",
      "publisher": "Mintlify",
      "verified": "2026-08-20",
      "keyFindings": "Mintlify auto-hosts llms.txt and llms-full.txt at both root and /.well-known/ paths; \"Page links in the llms.txt file include a .md extension so AI tools can fetch the Markdown version of each page directly\" (e.g. https://example.com/docs/api.md). Also advertises them via a standard HTTP Link header and an X-Llms-Txt header. Proves the per-page .md alternate is a deployed convention with header-level discovery, not a thought experiment."
    },
    {
      "id": "mintlify-llmstxt-docs",
      "title": "llms.txt — Mintlify documentation",
      "url": "https://www.mintlify.com/docs/ai/llmstxt",
      "type": "vendor-doc",
      "publisher": "Mintlify",
      "verified": "2026-08-20",
      "keyFindings": "Mintlify auto-generates llms.txt AND llms-full.txt for every hosted docs site (rolled out Nov 2024) — the single largest driver of llms.txt adoption. Serves at both /llms.txt and /.well-known/llms.txt, advertises locations via HTTP Link headers, and appends .md to page links 'so AI tools can fetch the Markdown version of each page directly.' Auto-generated llms.txt capped at 100,000 characters; no cap on llms-full.txt. Claims only vaguely that 'AI tools like ChatGPT and Claude can index and understand your documentation' — names no confirmed consumer."
    },
    {
      "id": "mintlify-no-headlink",
      "title": "llms.txt — Mintlify docs",
      "url": "https://mintlify.com/docs/ai/llmstxt",
      "publisher": "Mintlify",
      "type": "vendor-doc",
      "keyFindings": "Documents auto-hosting llms-full.txt at the project root; documents no <head> link element for discovery. mintlify.com/docs itself emits no llms link tag.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "mintlify-value-of-llmstxt",
      "title": "The value of llms.txt: Hype or real?",
      "url": "https://www.mintlify.com/blog/the-value-of-llms-txt-hype-or-real",
      "type": "article",
      "publisher": "Mintlify",
      "verified": "2026-08-20",
      "keyFindings": "The most-cited PRO-llms.txt claim, and it is weak. Dated 2025-05-09. Relays third-party data from Profound: 'models from Microsoft, OpenAI, and others are actively crawling and indexing both llms.txt and llms-full.txt files' and 'LLMs are accessing llms-full.txt even more frequently than the original llms.txt.' Provides NO sample size, NO methodology, NO frequency numbers, NO dataset access. Second-hand and unfalsifiable; also from a vendor that sells llms.txt generation."
    },
    {
      "id": "mistral-robots-docs",
      "title": "Mistral AI crawlers and robots.txt",
      "url": "https://docs.mistral.ai/robots/",
      "type": "vendor-doc",
      "publisher": "Mistral AI",
      "verified": "2026-08-20",
      "keyFindings": "Three tokens, all with +https://docs.mistral.ai/robots in the UA. MistralAI-User/1.0: user-initiated fetches for Vibe; robots.txt 'governs which sites user requests can access'; 'not used for crawling the web in any automatic fashion, nor to crawl content for generative AI training' — notably Mistral does NOT claim a robots.txt exemption for its user agent, unlike OpenAI/Perplexity/Meta. MistralAI-Index/1.0: search indexing, 'not used for generative AI training of any kind'. MistralAI-Training/1.0: training corpus collection; 'Webmasters can disallow this user agent in their robots.txt file.'"
    },
    {
      "id": "mozilla-readability-source",
      "title": "mozilla/readability Readability.js source",
      "url": "https://raw.githubusercontent.com/mozilla/readability/main/Readability.js",
      "type": "vendor-doc",
      "publisher": "Mozilla",
      "verified": "2026-08-20",
      "keyFindings": "The algorithm behind Firefox Reader Mode and most HTML→markdown reader pipelines. DEFAULT_TAGS_TO_SCORE = section,h2,h3,h4,h5,h6,p,td,pre. REGEXPS.unlikelyCandidates penalises class/id matching footer|header|menu|sidebar|related|social|comment|banner|breadcrumbs|sponsor…; okMaybeItsACandidate rescues and|article|body|column|content|main|shadow. UNLIKELY_ROLES = ['menu','menubar','complementary','navigation','alert','alertdialog','dialog'] — i.e. ARIA landmark roles are consulted directly to delete subtrees. _markDataTables uses dataTableDescendants = ['col','colgroup','tfoot','thead','th'] plus caption/summary to classify data vs layout tables, and _cleanConditionally exempts data tables ('if (tag === \"table\" && isDataTable(node)) return false'). Lists get a dedicated ratio test (listLength / innerText.length > 0.9). _getArticleTitle prefers the single <h1> when the <title> is ambiguous. Alt text is not scored; figures are only touched by _fixLazyImages."
    },
    {
      "id": "mozilla-standards-position-webmcp",
      "title": "Mozilla standards-positions — WebMCP (issue 1412)",
      "url": "https://github.com/mozilla/standards-positions/issues/1412",
      "type": "vendor-doc",
      "publisher": "Mozilla",
      "verified": "2026-08-20",
      "keyFindings": "Opened 2026-05-28, closed 2026-08-05. Labels: 'venue: W3C CG', 'position: neutral'. Mozilla is neutral — no commitment to implement."
    },
    {
      "id": "ms-copilot-api-plugins",
      "title": "Microsoft 365 Copilot — API plugins overview",
      "url": "https://learn.microsoft.com/en-us/microsoft-365-copilot/extensibility/overview-api-plugins",
      "type": "vendor-doc",
      "publisher": "Microsoft",
      "verified": "2026-08-21",
      "keyFindings": "Plugins \"interact with … REST APIs that have an OpenAPI description\", supplied inside a plugin manifest the developer packages and publishes."
    },
    {
      "id": "ms-copilot-openapi-guidance",
      "title": "Microsoft 365 Copilot — OpenAPI document guidance",
      "url": "https://learn.microsoft.com/en-us/microsoft-365-copilot/extensibility/openapi-document-guidance",
      "type": "vendor-doc",
      "publisher": "Microsoft",
      "verified": "2026-08-21",
      "keyFindings": "\"Operation IDs are unique identifiers for an operation in the API and are used by Copilot to create functions that are executed\" — one function per operation, selected by description."
    },
    {
      "id": "navigation-json-adoption",
      "title": "GitHub code search: navigation.json occurrences",
      "url": "https://github.com/search?q=%22navigation.json%22&type=code",
      "publisher": "GitHub",
      "type": "repo",
      "keyFindings": "(GitHub code-search query link — interactive only, result counts recorded at research time.) Substantive hits are documentation-toolchain build configs, not root-served agent manifests: ClickHouse/ClickHouse docs/products/cloud/navigation.json replicated across 10 locale directories, plus Nuxt/undocs/Mintlify-style docs configs. No shared schema matching the audited {name, items[{label,url,children}]} shape; no instance served at the site root as an agent signal.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "neon-catalog",
      "title": "Neon /.well-known/ai-catalog.json (live)",
      "url": "https://neon.com/.well-known/ai-catalog.json",
      "publisher": "Neon",
      "type": "vendor-doc",
      "keyFindings": "Live production manifest using specVersion/host/entries with 10 MCP server and skill entries. No owner/contact/lastUpdated fields.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "neon-catalog-fields",
      "title": "Neon live manifest field audit",
      "url": "https://neon.com/.well-known/ai-catalog.json",
      "publisher": "Neon",
      "type": "vendor-doc",
      "keyFindings": "Confirms absence of owner/contact/lastUpdated in a production manifest; uses specVersion/host/entries.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "neon-catalog-urls",
      "title": "Neon live manifest entries",
      "url": "https://neon.com/.well-known/ai-catalog.json",
      "publisher": "Neon",
      "type": "vendor-doc",
      "keyFindings": "10 entries pointing at MCP servers and skills — operational endpoints an agent will call directly, making liveness consequential.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "neon-link-header",
      "title": "neon.com HTTP Link header",
      "url": "https://neon.com/",
      "publisher": "Neon",
      "type": "vendor-doc",
      "keyFindings": "Link header includes </.well-known/ai-catalog.json>; rel=\"ai-catalog\" alongside rel=\"api-catalog\", rel=\"llms-txt\", rel=\"mcp-server-card\" — advertisement happens via header, which the audit does not inspect at all.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "nextjs-large-page-data",
      "title": "Large Page Data (Next.js error reference)",
      "url": "https://nextjs.org/docs/messages/large-page-data",
      "type": "vendor-doc",
      "publisher": "Vercel / Next.js",
      "verified": "2026-08-20",
      "keyFindings": "Warns when a page ships > 128 kB of serialized __NEXT_DATA__ JSON; states \"The serialized data is inlined in every HTML response, increasing the document size\"; threshold configurable via experimental.largePageDataBytes. Gives a vendor-sanctioned hard numeric threshold for inlined hydration state in the HTML document."
    },
    {
      "id": "nlweb-github",
      "title": "NLWeb — reference implementation",
      "url": "https://github.com/nlweb-ai/NLWeb",
      "type": "spec",
      "publisher": "Microsoft / nlweb-ai (MIT)",
      "verified": "2026-08-20",
      "keyFindings": "Positions \"Schema.org and related semi-structured formats like RSS — used by over 100 million websites\" as the semantic layer; works with \"existing markup on sites with structured lists (e.g., products, recipes, attractions, reviews)\". \"Every NLWeb instance also acts as an MCP server ... and supports a core method, ask, which allows a natural language question to be posed\", with responses expressed in schema.org vocabulary. This is the strongest existing evidence of an AI-agent protocol consuming site schema.org directly."
    },
    {
      "id": "nlweb-repo",
      "title": "nlweb-ai/NLWeb — reference implementation",
      "url": "https://github.com/nlweb-ai/NLWeb",
      "type": "vendor-doc",
      "publisher": "nlweb-ai (originally Microsoft)",
      "verified": "2026-08-20",
      "keyFindings": "6,249 stars, last push 2026-08-11, MIT licensed, ~1,184 commits. The repo has been TRANSFERRED out of the Microsoft org: api.github.com/repos/microsoft/NLWeb returns HTTP 301 → repositories/974473130 (nlweb-ai/NLWeb). README: 'It natively supports MCP'; '/ask' is a core MCP method; 'In short, NLWeb is to MCP/A2A what HTML is to HTTP.' No .well-known discovery path is defined anywhere in the README."
    },
    {
      "id": "nlweb-repo-actions",
      "title": "NLWeb repository grep for action types",
      "url": "https://github.com/nlweb-ai/NLWeb",
      "publisher": "Microsoft / nlweb-ai",
      "type": "repo",
      "keyFindings": "Cloned and grepped: 'ConfirmAction' 0 files, 'ReserveAction' 0 files, 'OrderAction' 0 files, 'potentialAction' 0 files. Microsoft's schema.org-native agent framework ingests entity types (Recipe: 33 files, Product, Brand), not Actions.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "nlweb-repo-howto",
      "title": "NLWeb repository grep for HowTo",
      "url": "https://github.com/nlweb-ai/NLWeb",
      "publisher": "Microsoft / nlweb-ai",
      "type": "repo",
      "keyFindings": "Cloned and grepped: 'HowTo' appears in 0 files across .py/.md/.json/.ts, while 'Recipe' appears in 33. Microsoft's schema.org-based agent framework does not consume HowTo.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "oai-issue-1851-discovery",
      "title": "OAI/OpenAPI-Specification issue 1851 — RFC 5785 conventions & suggested OpenAPI document locations",
      "url": "https://github.com/OAI/OpenAPI-Specification/issues/1851",
      "type": "draft-spec",
      "publisher": "OpenAPI Initiative",
      "verified": "2026-08-20",
      "keyFindings": "Opened 2019-02-27, label 'discovery', CLOSED as completed (last activity 2025-08-19). Asked whether OAS should recommend a `/.well-known/` URI for OpenAPI documents. OAS 3.2.0 ships with NO normative well-known location for the OpenAPI document itself; its only 'well-known' reference is to OpenID Connect Discovery inside the security-scheme object."
    },
    {
      "id": "observation-reduction-paper",
      "title": "Read More, Think More: Revisiting Observation Reduction for Web Agents",
      "url": "https://arxiv.org/abs/2604.01535",
      "type": "study",
      "publisher": "arXiv (Enomoto, Obara, Zhang, Oyamada, 2026)",
      "verified": "2026-08-20",
      "keyFindings": "MAJOR COUNTER-EVIDENCE plus the best token quantification available. Measured input cost per step: HTML ≈ 56,653 tokens vs accessibility tree ≈ 6,720 (~8.4×). But 'compact observations (accessibility trees) are preferable for lower-capability models, while detailed observations (HTML) are advantageous for higher-capability models': Claude Sonnet 4.6 +14.6pp and GPT-5.1 (high reasoning) +17.5pp when given HTML instead of the a11y tree; GPT-OSS-20B -18.8pp and Qwen3-VL-30B-Thinking -7.1pp. Mechanism: strong models 'exploit layout information in HTML for better action grounding' (including CSS z-index to avoid intercepted clicks); weak models 'suffer from increased hallucination under longer inputs'. So the a11y tree is not universally the interface, and larger DOMs are not monotonically harmful."
    },
    {
      "id": "ogp-me-spec",
      "title": "The Open Graph protocol",
      "url": "https://ogp.me/",
      "type": "spec",
      "publisher": "Open Graph protocol (originated at Facebook), Open Web Foundation Agreement v0.9",
      "verified": "2026-08-21",
      "keyFindings": "Defines the four required properties (og:title, og:type, og:image, og:url) and optional og:description, og:site_name, og:locale. Governed under Open Web Foundation Agreement 0.9 — not a W3C/IETF standard. Names social platforms and search engines as consumers; no AI/answer-engine consumer is named anywhere in the spec."
    },
    {
      "id": "ogp-types",
      "title": "Open Graph protocol — types",
      "url": "https://ogp.me/#types",
      "type": "spec",
      "publisher": "Open Graph protocol",
      "verified": "2026-08-21",
      "keyFindings": "Defines the type vocabulary an `og:type` value must come from: `article`, `book`, `profile`, `website`, `music.*`, `video.*`."
    },
    {
      "id": "oncrawl-ai-bot-logs",
      "title": "What AI bots are really doing on your site (production server-log analysis)",
      "url": "https://www.oncrawl.com/ai/what-ai-bots-really-doing-your-site/",
      "type": "study",
      "publisher": "Oncrawl",
      "verified": "2026-08-20",
      "keyFindings": "Documents two discovery pathologies with named crawlers: 'OAI-SearchBot gets lost in pagination' — a travel site saw it crawl non-existent pagination URLs with no site change; and 'ChatGPT inventing URLs' on Boulanger.com, where 988 ChatGPT-User requests returned 404 concentrated over a few hours because the model hallucinated product-listing paths. Site-level AI bot traffic share 0.1–1%, median 0.4%."
    },
    {
      "id": "openai-apps-sdk-deploy",
      "title": "OpenAI Apps SDK — Deploy",
      "url": "https://developers.openai.com/apps-sdk/deploy/",
      "type": "vendor-doc",
      "publisher": "OpenAI",
      "verified": "2026-08-20",
      "keyFindings": "Connection is via ChatGPT developer mode (paste URL) for testing, and for production: 'deploy the MCP server at a stable, publicly reachable HTTPS endpoint' for 'public plugin submission', with 'plugin review and domain verification'. No automatic .well-known discovery is described."
    },
    {
      "id": "openai-apps-sdk-mcp",
      "title": "OpenAI Apps SDK — current third-party extensibility, built on MCP",
      "url": "https://developers.openai.com/apps-sdk/",
      "publisher": "OpenAI",
      "type": "vendor-doc",
      "keyFindings": "OpenAI's current mechanism for third-party apps in ChatGPT uses the Model Context Protocol: docs describe building an MCP server to 'add live data and controlled tools' and give an app 'tools and access to external systems.' No mention of ai-plugin.json or plugin manifests anywhere — confirming the successor is MCP, not a revived manifest.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "openai-apps-sdk-mcp-server",
      "title": "OpenAI Apps SDK — MCP server concepts",
      "url": "https://developers.openai.com/apps-sdk/concepts/mcp-server",
      "type": "vendor-doc",
      "publisher": "OpenAI",
      "verified": "2026-08-20",
      "keyFindings": "ChatGPT apps are MCP servers ('Deploy production MCP servers at stable HTTPS endpoints using the streamable HTTP transport'). Confirms ChatGPT as a documented MCP consumer, but the page defers all discovery/registration mechanics to submission, not to site crawling."
    },
    {
      "id": "openai-apps-sdk-security",
      "title": "Security & Privacy — Apps SDK",
      "url": "https://developers.openai.com/apps-sdk/guides/security-privacy",
      "type": "vendor-doc",
      "publisher": "OpenAI",
      "verified": "2026-08-20",
      "keyFindings": "'Widgets run inside an isolated iframe with a strict Content Security Policy.' Nested frames are unavailable by default and must be opted into via _meta.ui.csp.frameDomains; connect_domains maps to connect-src and resource_domains to style-src/img-src/font-src/media-src. Widgets cannot use window.alert/prompt/confirm or navigator.clipboard. Shows that CSP/CORS constraints in the AI-agent world are imposed BY the agent host on the widget, not read FROM the publisher's site."
    },
    {
      "id": "openai-bots",
      "title": "OpenAI Bots / Crawler documentation",
      "url": "https://developers.openai.com/api/docs/bots",
      "type": "vendor-doc",
      "publisher": "OpenAI",
      "verified": "2026-08-21",
      "keyFindings": "Four distinct user agents with separate robots.txt tokens and separate published IP-range files: OAI-SearchBot (surfaces sites in ChatGPT search — https://openai.com/searchbot.json), OAI-AdsBot (validates ad landing pages — https://openai.com/adsbot.json), GPTBot (model training — https://openai.com/gptbot.json), ChatGPT-User (user-initiated actions: web visits and GPT Actions — https://openai.com/chatgpt-user.json). ChatGPT-User is the agent that fetches on a shopper's behalf. Crucially these are separately controllable: blocking GPTBot does not block OAI-SearchBot or ChatGPT-User, and vice versa."
    },
    {
      "id": "openai-bots-doc",
      "title": "Overview of OpenAI Crawlers",
      "url": "https://developers.openai.com/api/docs/bots",
      "publisher": "OpenAI",
      "type": "vendor-doc",
      "keyFindings": "Documents OAI-SearchBot, OAI-AdsBot, GPTBot and ChatGPT-User. Control surface is exclusively robots.txt plus published IP-range JSON files. No mention of any HTTP response header, security header, Referrer-Policy, Permissions-Policy, security.txt, JS rendering, page speed, or preconnect.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "openai-bots-docs",
      "title": "OpenAI Bots / Crawlers documentation",
      "url": "https://developers.openai.com/api/docs/bots",
      "type": "vendor-doc",
      "publisher": "OpenAI",
      "verified": "2026-08-21",
      "keyFindings": "Four documented bots as of Aug 2026 (note: platform.openai.com/docs/bots now 301s here). GPTBot/1.4 (+https://openai.com/gptbot, IPs at openai.com/gptbot.json): 'Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models.' OAI-SearchBot/1.4 (+https://openai.com/searchbot, IPs at openai.com/searchbot.json): 'Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers.' ChatGPT-User/1.0 (+https://openai.com/bot, IPs at openai.com/chatgpt-user.json): 'Because these actions are initiated by a user, robots.txt rules may not apply.' OAI-AdsBot/1.0 (+https://openai.com/adsbot) — new; validates safety of pages submitted as ChatGPT ads, 'only visits pages submitted as ads'. ChatGPT-Agent is NOT listed on this page."
    },
    {
      "id": "openai-bots-framework",
      "title": "OpenAI crawlers and user agents",
      "url": "https://developers.openai.com/api/docs/bots",
      "publisher": "OpenAI",
      "type": "vendor-doc",
      "verified": "2026-08-21",
      "keyFindings": "No statement about JavaScript execution, rendering, or frameworks for GPTBot, OAI-SearchBot or ChatGPT-User. Documentation is limited to user agents, IP ranges, robots.txt and use cases — so framework identity cannot be mapped to any disclosed consumer capability.",
      "accessed": "2026-08-21"
    },
    {
      "id": "openai-bots-overview",
      "title": "Overview of OpenAI Crawlers",
      "url": "https://developers.openai.com/api/docs/bots",
      "type": "vendor-doc",
      "publisher": "OpenAI",
      "verified": "2026-08-21",
      "keyFindings": "Documents OAI-SearchBot ('used to surface websites in search results in ChatGPT's search features'), OAI-AdsBot, GPTBot, ChatGPT-User. States ~24h robots.txt propagation. COUNTER-EVIDENCE: contains NO mention of sitemaps, RSS/Atom feeds, canonical URLs, rel=nofollow, breadcrumbs, pagination, or any URL-discovery mechanism whatsoever."
    },
    {
      "id": "openai-bots-pagination",
      "title": "OpenAI crawlers and user agents",
      "url": "https://developers.openai.com/api/docs/bots",
      "publisher": "OpenAI",
      "type": "vendor-doc",
      "verified": "2026-08-21",
      "keyFindings": "Covers user-agent strings, published IP ranges, robots.txt configuration and per-bot use cases. Contains no mention of link relation tags, pagination handling, or JavaScript rendering.",
      "accessed": "2026-08-21"
    },
    {
      "id": "openai-chatgpt-search-help",
      "title": "ChatGPT Search / Searching the web with ChatGPT",
      "url": "https://help.openai.com/en/articles/9237897-chatgpt-search",
      "type": "vendor-doc",
      "publisher": "OpenAI Help Center",
      "verified": "2026-08-20",
      "keyFindings": "Describes query rewriting, third-party search partners, location handling and inline citations. Contains no reference to schema.org, JSON-LD or structured data as an input to retrieval or citation selection. Listed as counter-evidence (absence of documentation); URL surfaced via search but body fetch was not completed, so marked unverified."
    },
    {
      "id": "openai-commerce",
      "title": "OpenAI Commerce (overview)",
      "url": "https://developers.openai.com/commerce/",
      "type": "vendor-doc",
      "publisher": "OpenAI",
      "verified": "2026-08-20",
      "keyFindings": "Three feed delivery paths: API create/retrieve/upsert of product + promotion feed data, SFTP file upload, and legacy file upload. Feed conformance is what lets 'ChatGPT accurately index and display your products with up-to-date price and availability.' Checkout requires the Agentic Checkout Spec. Page does not document schema.org markup as an alternative ingestion path."
    },
    {
      "id": "openai-commerce-docs-index",
      "title": "Agentic Commerce documentation index",
      "url": "https://developers.openai.com/commerce/",
      "type": "vendor-doc",
      "publisher": "OpenAI",
      "verified": "2026-08-20",
      "keyFindings": "Full section listing (Guides: key concepts, best practices, production; Specs: Agentic Checkout, Delegated Payment, Feeds, File Upload, Onboarding, Products, Promotions, Product Feed Spec). No page in the index references schema.org, JSON-LD, or on-page markup. Confirms absence of documentation, not just omission on one page."
    },
    {
      "id": "openai-commerce-feed-spec",
      "title": "Product Feed Spec — Agentic Commerce | OpenAI Developers",
      "url": "https://developers.openai.com/commerce/specs/feed",
      "type": "vendor-doc",
      "publisher": "OpenAI",
      "verified": "2026-08-21",
      "keyFindings": "Feed fields include 'url' = 'Product detail page URL' (example https://example.com/product/SKU12345) which 'must resolve with HTTP 200; HTTPS preferred', and 'seller_url' derived from the link's scheme and authority. There is NO add-to-cart, checkout, or deep-link field. Submission is by upload of UTF-8 TSV/CSV (optionally gzipped); explicitly: 'JSON, spreadsheet, XML, RSS, and Atom sources are not part of this compatibility path.'"
    },
    {
      "id": "openai-commerce-index",
      "title": "OpenAI Agentic Commerce documentation index",
      "url": "https://developers.openai.com/commerce/",
      "type": "vendor-doc",
      "publisher": "OpenAI",
      "verified": "2026-08-20",
      "keyFindings": "Four specs: Product Feed (specs/spec.md), Agentic Checkout (specs/checkout.md), Delegated Payment (specs/payment.md), Feeds API (specs/api/feeds.md). Feed delivery via API (products, promotions endpoints) or File Upload/SFTP. No public feed-URL convention and no discovery file is defined. Feeds let ChatGPT 'accurately index and display your products with up-to-date price and availability'."
    },
    {
      "id": "openai-commerce-key-concepts",
      "title": "Key concepts — Agentic Commerce | OpenAI Developers",
      "url": "https://developers.openai.com/commerce/guides/key-concepts",
      "type": "vendor-doc",
      "publisher": "OpenAI",
      "verified": "2026-08-20",
      "keyFindings": "Merchants supply 'a secure, regularly refreshed feed (CSV or JSON)' via initial sample validation plus daily snapshots. Instant Checkout renders inside ChatGPT's UI while ChatGPT calls the merchant's ACP endpoints; 'the actual checkout state and payment processing occurs on the merchant's systems.' No crawling, sitemap, or automatic site-discovery mechanism is described anywhere in the product-discovery path."
    },
    {
      "id": "openai-community-plugins-closure",
      "title": "Plugin Store and New Chats With Plugins — Closed March 19 2024",
      "url": "https://community.openai.com/t/plugin-store-and-new-chats-with-plugins-closed-march-19-2024/689877",
      "type": "article",
      "publisher": "OpenAI Developer Community",
      "verified": "2026-08-20",
      "keyFindings": "Quotes OpenAI's help-centre wording: 'On March 19, 2024, you will no longer be able to install new plugins or create new conversations with existing plugins. You will be able to continue existing conversations until April 9, 2024.' Note: OpenAI's own help.openai.com article (8988022) and openai.com return HTTP 403 to automated fetches, so this forum thread is the fetchable carrier of the primary quote."
    },
    {
      "id": "openai-computer-use-guide",
      "title": "Computer use — OpenAI API guide",
      "url": "https://developers.openai.com/api/docs/guides/tools-computer-use",
      "type": "vendor-doc",
      "publisher": "OpenAI",
      "verified": "2026-08-21",
      "keyFindings": "COUNTER-EVIDENCE. Screenshots only: 'The model looks at the current UI through a screenshot, returns actions such as clicks, typing, or scrolling, and your harness executes those actions.' Inputs are base64 PNG screenshots with detail:'original'. No mention of DOM, accessibility tree, or semantic HTML anywhere in the input format."
    },
    {
      "id": "openai-connectors-mcp",
      "title": "Tools: Connectors and MCP",
      "url": "https://developers.openai.com/api/docs/guides/tools-connectors-mcp",
      "publisher": "OpenAI",
      "type": "vendor-doc",
      "keyFindings": "Capability exposure to OpenAI models is via remote MCP servers (server_url) and OpenAI-maintained connectors (connector_id); tools are discovered through explicit server configuration in API requests, not by parsing HTML. No data-* attribute or any HTML-based mechanism is documented, contradicting the audit's claim that ChatGPT Browse reads data-action.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "openai-crawlers-doc",
      "title": "Overview of OpenAI Crawlers",
      "url": "https://developers.openai.com/api/docs/bots",
      "type": "vendor-doc",
      "publisher": "OpenAI",
      "verified": "2026-08-21",
      "keyFindings": "NEGATIVE/ABSENCE EVIDENCE (identified via search; not fetched, so marked unverified). OpenAI's crawler documentation governs GPTBot, OAI-SearchBot and ChatGPT-User exclusively through robots.txt and does not mention llms.txt. OpenAI has published no statement that any of its systems parse or prioritize llms.txt. Separately, OpenAI's own developer docs DO publish /llms.txt and serve .md variants by URL suffix."
    },
    {
      "id": "openai-crawlers-docs",
      "title": "Overview of OpenAI Crawlers",
      "url": "https://developers.openai.com/api/docs/bots",
      "type": "vendor-doc",
      "publisher": "OpenAI",
      "verified": "2026-08-21",
      "keyFindings": "Documents GPTBot (training), OAI-SearchBot (ChatGPT search), ChatGPT-User (user-initiated), OAI-AdsBot, with full UA strings and IP-range JSON files (openai.com/gptbot.json, /searchbot.json, /chatgpt-user.json, /adsbot.json). NOTABLE ABSENCE: the page says nothing about JavaScript rendering, HTTPS requirements, caching/conditional requests, content types, response timeouts, or sitemaps. Primary counter-evidence for any signal claiming OpenAI documents those behaviors."
    },
    {
      "id": "openai-cua-announcement",
      "title": "Computer-Using Agent (CUA)",
      "url": "https://openai.com/index/computer-using-agent/",
      "type": "vendor-doc",
      "publisher": "OpenAI",
      "verified": "2026-08-20",
      "keyFindings": "Returned HTTP 403 to automated fetch, so NOT verified — cited only as secondary corroboration of the OpenAI API guide, which was verified. Widely reported as stating CUA processes raw pixel data from screenshots and uses a virtual mouse/keyboard 'without needing specialized APIs'. Rely on openai-computer-use-guide instead."
    },
    {
      "id": "openai-deprecations",
      "title": "OpenAI API — Deprecations",
      "url": "https://developers.openai.com/api/docs/deprecations",
      "type": "vendor-doc",
      "publisher": "OpenAI",
      "verified": "2026-08-20",
      "keyFindings": "COUNTER-EVIDENCE by absence: the current deprecations page lists models, Assistants API, Videos API, Evals, Agent Builder and Reusable Prompts — and contains NO entry for ChatGPT plugins or ai-plugin.json, because plugins were removed from the platform entirely long before this page's current scope."
    },
    {
      "id": "openai-feed-spec",
      "title": "OpenAI Commerce — Product Feed Spec",
      "url": "https://developers.openai.com/commerce/specs/feed/",
      "type": "vendor-doc",
      "publisher": "OpenAI",
      "verified": "2026-08-21",
      "keyFindings": "Required per item: item_id (stable, unique per variant, max 100 chars), title, description (max 5000, plain text only), link/url (must resolve HTTP 200), image_url (JPEG/PNG), price with ISO 4217 currency, availability from lowercase enum {in_stock,out_of_stock,pre_order,backorder,unknown}, brand (max 70), seller_name, target_countries (ISO 3166-1 alpha-2), is_eligible_search (bool), is_eligible_checkout (bool, requires is_eligible_search=true). Optional/recommended: gtin (8-14 digits, no dashes), mpn (max 70), condition, item_group_id (stable across variants). sale_price must be <= price. Accepted formats are delimited files only (TSV/CSV/.txt + gzip); 'JSON, spreadsheet, XML, RSS, and Atom sources are not part of this compatibility path.' No documented refresh-cadence requirement."
    },
    {
      "id": "openai-feed-spec-confirm",
      "title": "Product feed specification",
      "url": "https://developers.openai.com/commerce/specs/feed",
      "publisher": "OpenAI",
      "type": "vendor-doc",
      "keyFindings": "Requires tab- or comma-delimited flat files; 'JSON, spreadsheet, XML, RSS, and Atom sources are not part of this compatibility path.' Confirms OpenAI's commerce stack does not consume JSON-LD of any kind.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "openai-function-calling",
      "title": "OpenAI — function calling guide",
      "url": "https://developers.openai.com/api/docs/guides/function-calling",
      "type": "vendor-doc",
      "publisher": "OpenAI",
      "verified": "2026-08-21",
      "keyFindings": "A function's `parameters` are \"defined by a JSON schema\", supporting \"property types, enums, descriptions, nested objects\"."
    },
    {
      "id": "openai-getstarted",
      "title": "OpenAI Commerce: Get Started (feed onboarding)",
      "url": "https://developers.openai.com/commerce/guides/get-started",
      "type": "vendor-doc",
      "publisher": "OpenAI",
      "verified": "2026-08-20",
      "keyFindings": "Product feed onboarding in ChatGPT is currently limited to approved partners via an application form. Confirms feed ingestion is push/upload-based (file upload or API), not a merchant-hosted public URL that OpenAI discovers — reinforcing that site-side audit checks must test the DATA a feed would be built from rather than the feed transport."
    },
    {
      "id": "openai-gpt-actions-extensions",
      "title": "GPT Actions — Introduction (checked for AI OpenAPI extensions)",
      "url": "https://developers.openai.com/api/docs/actions/introduction",
      "publisher": "OpenAI",
      "type": "vendor-doc",
      "keyFindings": "OpenAI's Actions documentation describes driving actions from an OpenAPI spec via Function Calling but makes no mention of x-ai-instructions. The most plausible vendor consumer does not read the field.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "openai-gpt-actions-intro",
      "title": "GPT Actions — Introduction",
      "url": "https://developers.openai.com/api/docs/actions/introduction",
      "publisher": "OpenAI",
      "type": "vendor-doc",
      "keyFindings": "The successor documentation to plugins. Explains Actions via Function Calling with an OpenAPI example; contains no reference to ai-plugin.json, plugin manifests, or the /.well-known/ path.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "openai-gpt-actions-openapi",
      "title": "OpenAI — Getting started with GPT Actions",
      "url": "https://developers.openai.com/api/docs/actions/getting-started",
      "type": "vendor-doc",
      "publisher": "OpenAI",
      "verified": "2026-08-21",
      "keyFindings": "'A GPT Action requires an Open API schema to describe the parameters of the API call, which is a standard for describing APIs.' This is a documented consumer of the OpenAPI FORMAT. The page contains no mention of ai-plugin.json and no automatic discovery — the schema is pasted or fetched from a URL the builder supplies."
    },
    {
      "id": "openai-help-plugins-winddown",
      "title": "Winding down the ChatGPT plugins beta (OpenAI Help Center)",
      "url": "https://help.openai.com/en/articles/8988022-winding-down-the-chatgpt-plugins-beta",
      "publisher": "OpenAI",
      "type": "announcement",
      "keyFindings": "(Resolves; returns 403 to non-browser clients — page exists, bot-blocked.) Could not be verified — the help center returned HTTP 403 to automated fetching, and web.archive.org is not fetchable from this environment. No claim is based on this document; the discontinuation is instead established by the archived openai/plugins-quickstart repo and the 404'd manifest docs, both verified directly.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "openai-plugin-manifest-docs-404",
      "title": "OpenAI plugin manifest documentation — removed (301 to 404)",
      "url": "https://platform.openai.com/docs/plugins/getting-started/plugin-manifest",
      "publisher": "OpenAI",
      "type": "vendor-doc",
      "keyFindings": "curl -L shows a single 301 to https://developers.openai.com/api/docs/actions, which returns HTTP 404 (final_code=404). The plugins docs root /docs/plugins/introduction lands on the same 404. This is the exact docsUrl the audit ships, so its remediation link is broken; OpenAI no longer publishes any ai-plugin.json specification.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "openai-plugins-quickstart-archived",
      "title": "openai/plugins-quickstart (ARCHIVED) — official ChatGPT plugin quickstart",
      "url": "https://github.com/openai/plugins-quickstart",
      "publisher": "OpenAI",
      "type": "repo",
      "keyFindings": "GitHub API confirms archived=true, pushed_at=2024-01-30T23:23:11Z, 4,236 stars. README states verbatim: 'Plugins have been superseded by GPTs, learn more about creating a GPT with actions.' This is OpenAI's own first-party notice that the plugins program (and its ai-plugin.json manifest) is over.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "openai-product-feed-spec",
      "title": "Product Feed Spec — Products (Agentic Commerce)",
      "url": "https://developers.openai.com/commerce/specs/file-upload/products",
      "type": "spec",
      "publisher": "OpenAI",
      "verified": "2026-08-20",
      "keyFindings": "Documented consumer of rating/review data by a named AI surface (ChatGPT shopping). Review fields: review_count (Optional, 'Number of product reviews'), star_rating (Optional, average review score 0–5), store_review_count (Optional, 'Number of brand or store reviews'), store_star_rating (Optional, average store rating 0–5), reviews (Recommended — entries with title, content, ratings), q_and_a (Recommended — FAQ question/answer pairs). Brand/seller: brand (REQUIRED, max 70 chars), seller_name (Required for display; OpenAI replaces uploaded values with the registered merchant name), marketplace_seller (Optional). Trust/returns: accepts_returns, return_deadline_in_days, accepts_exchanges, return_policy (all Optional); seller_privacy_policy and seller_tos required if checkout is enabled. Note: this is a pushed merchant FEED over allow-listed HTTPS, not on-page schema scraping."
    },
    {
      "id": "openai-publishers-faq",
      "title": "Publishers and Developers - FAQ",
      "url": "https://help.openai.com/en/articles/12627856-publishers-and-developers-faq",
      "type": "vendor-doc",
      "publisher": "OpenAI Help Center",
      "verified": "2026-08-20",
      "keyFindings": "URL exists and is indexed but returns HTTP 403 to non-browser clients, so I could not verify contents first-hand. Search-surfaced quotation attributed to this page: \"In order for our crawler to read a meta tag, it must be allowed to crawl the relevant page(s).\" Treat as unverified secondary until fetched from a browser session — it is the only reported OpenAI statement acknowledging meta-tag reading at all."
    },
    {
      "id": "openai-searchbot-ips",
      "title": "OAI-SearchBot published IP ranges",
      "url": "https://openai.com/searchbot.json",
      "type": "dataset",
      "publisher": "OpenAI",
      "verified": "2026-08-20",
      "keyFindings": "Live JSON with creationTime (2026-01-02T11:00:00.000000) and a prefixes[] array of 35 IPv4 CIDR blocks (/24 to /28), e.g. 104.210.140.128/28 and 172.182.193.224/28. Machine-consumable, so an auditor can both spoof the documented UA and tell a merchant exactly which ranges to allowlist at the WAF."
    },
    {
      "id": "openapi-31-spec",
      "title": "OpenAPI Specification 3.1.0",
      "url": "https://spec.openapis.org/oas/v3.1.0.html",
      "type": "spec",
      "publisher": "OpenAPI Initiative",
      "verified": "2026-08-21",
      "keyFindings": "Defines `summary` as \"A short summary of what the operation does\" alongside `description`, \"A verbose explanation\" — so a consumer reading \"the description\" may mean either key."
    },
    {
      "id": "openapi-discovery-conventions",
      "title": "openapisearch — OpenAPI discovery conventions",
      "url": "https://github.com/janwilmake/openapisearch/blob/main/docs/openapi-discovery.md",
      "type": "article",
      "publisher": "janwilmake (community)",
      "verified": "2026-08-20",
      "keyFindings": "Lists the de-facto community convention paths: `/openapi.json`, `/openapi.yaml`, and `/.well-known/openapi` (as an RFC 8615-style redirect). Provides NO adoption statistics and NO empirical support for the claim that publishing at these paths makes an API 'discoverable to the widest range of tools'."
    },
    {
      "id": "openapi-extensions-registry",
      "title": "OpenAPI Initiative — Specification Extensions Registry",
      "url": "https://spec.openapis.org/registry/extension/",
      "publisher": "OpenAPI Initiative",
      "type": "spec",
      "keyFindings": "Authoritative registry of OpenAPI specification extensions. x-ai-instructions is NOT registered. Of the 36 registered extensions (x-agent-trust, x-codeSamples, x-data-classification, x-jsonld-context/type, x-jsonschema-*, x-oai-*, x-sensitive-data, x-twitter), none use an x-ai-* prefix and none target AI/LLM guidance.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "originality-ai-bot-blocking",
      "title": "AI Bot Blocking — live tracker of top 1000 sites",
      "url": "https://originality.ai/ai-bot-blocking",
      "type": "dataset",
      "publisher": "Originality.AI",
      "verified": "2026-08-20",
      "keyFindings": "URL resolves. Live dashboard tracking which of the top 1,000 websites by traffic block AI bots. Static page exposes little: per-bot blocking rates and methodology are rendered client-side and were not machine-extractable. One cited figure: crawler restriction caused a 28–45% decline in accessible training corpora, with 45% of C4 now restricted. Treat headline percentages as unverified until read from the interactive dashboard."
    },
    {
      "id": "originality-noai-adoption",
      "title": "Noai and noimageai Tag Adoption: Study and Live Dashboard",
      "url": "https://originality.ai/blog/noai-noimageai-adoption-dashboard",
      "type": "dataset",
      "publisher": "Originality.AI",
      "verified": "2026-08-20",
      "keyFindings": "Adoption data as of June 2026: 88,000+ domains carrying a noai or noimageai tag in their testing pool; 87.8% of adopters use the meta-tag placement, 26.7% the X-Robots-Tag header, 14.5% both. Meta-tag adoption grew 61,403 -> 77,645 domains May->June 2026 (+26.5%) while header adoption was flat (+2%). Explicit counter-finding: \"Major AI companies point elsewhere ... rather than honoring the noai meta tag specifically\", noting \"Google's own robots meta documentation lists the directives it recognizes ... Noai is not on that list.\" Methodology (pool composition, sampling) is not disclosed — treat magnitudes as indicative."
    },
    {
      "id": "oss-github",
      "title": "GitHub repository census — llms.txt / MCP / agent-readiness auditing tools",
      "url": "https://github.com/search?q=llms.txt+validator&type=repositories",
      "type": "dataset",
      "publisher": "GitHub (queried via GitHub REST search API)",
      "verified": "2026-08-20",
      "keyFindings": "Enumerated via gh api search/repositories. The field is generators, not auditors: AnswerDotAI/llms-txt (2575*, the spec itself), firecrawl/llmstxt-generator (537*), delucis/starlight-llms-txt (109*), thedaviddias/mcp-llms-txt-explorer (76*). Auditor-shaped projects are all <5 stars: hanselhansel/context-cli (robots+llms.txt+Schema.org+content density, 0-100), agentmarkup/agentmarkup (22*, build-time generation+validation), portdeveloper/llms-txt-check (1*, validates llms.txt against what the site actually serves), mikiships/agent-trust-scan (1*, A2A+MCP+llms.txt endpoint validation), abhi725/growth-mcp (1*), JerryZhi/AI-Crawler-Detector (5*, detects server-side AI crawler blocking beyond robots.txt), arturseo-geo/mcp-crawl-parity (1*, Googlebot vs AI crawler parity from Nginx logs). No project combines active differential fetching with robots.txt policy reconciliation."
    },
    {
      "id": "otterly",
      "title": "Otterly.ai",
      "url": "https://otterly.ai/",
      "type": "vendor-doc",
      "publisher": "Otterly.ai",
      "verified": "2026-08-20",
      "keyFindings": "Monitors 7 engines (ChatGPT, Google AI Overviews, AI Mode, Gemini, Perplexity, Copilot, Claude via API). Features: AI Prompt Research, AI Search Analytics, Content Audit (advertises 'crawlability checks' and a predictive citation score), GEO Optimization. Marketing names no specific technical check — no robots.txt, llms.txt, schema or bot-access validation is documented."
    },
    {
      "id": "otterly-homepage",
      "title": "Otterly.AI",
      "url": "https://otterly.ai/",
      "type": "vendor-doc",
      "publisher": "Otterly.AI",
      "verified": "2026-08-20",
      "keyFindings": "Hybrid: prompt-based citation tracking (Share of AI Voice, average position, sentiment across ChatGPT, Perplexity, AI Overviews, Gemini, Copilot, Claude, AI Mode) plus a 'Content Audit' doing crawlability checks and 'citation potential' prediction. Methodology is described only at the level of 'users define prompts, we run them across engines'. Evidence offered is client case studies (Bacula Systems, Instant Commerce, ChatArmin); no methodological validation, no per-check sourcing."
    },
    {
      "id": "otterly-llmstxt-experiment",
      "title": "llms.txt and AI Visibility: Results from OtterlyAI's GEO Study",
      "url": "https://otterly.ai/blog/the-llms-txt-experiment/",
      "type": "study",
      "publisher": "Otterly.ai",
      "verified": "2026-08-20",
      "keyFindings": "90-day continuous server-log study of AI/LLM user agents. 62,100+ AI bot requests total; exactly 84 hit /llms.txt = ~0.1%. The file drew ~3x fewer AI visits than an average content page on the same domain. Conclusion: 'llms.txt is not a significant driver of AI crawl behavior'; agents discover content 'via standard web pages and existing site structure, not via llms.txt.' Does not break down which bots made the 84 requests."
    },
    {
      "id": "otterly-markdown-vs-html",
      "title": "GEO Experiment: Markdown vs. HTML, Which Format Do AI Crawlers Prefer?",
      "url": "https://otterly.ai/blog/geo-experiment-html-vs-markdown/",
      "type": "study",
      "publisher": "Otterly.ai",
      "verified": "2026-08-20",
      "keyFindings": "KEY COUNTER-EVIDENCE for markdown alternates. 14-day controlled test, two scenarios (existing HTML + new .md; new HTML + .md published together), equal footer-link discoverability. HTML test pages: 137 crawler visits. Markdown files: 0 visits, 0 citations across all tracked platforms. 'If AI bots never visit a .md file, that file never enters the retrieval pipeline.' Note the scope limit: this measures indexing crawlers, not interactive coding agents, and tests .md URLs rather than Accept-header negotiation."
    },
    {
      "id": "palewire-news-homepages-robotstxt",
      "title": "News Homepages — AI crawler robots.txt blocking archive",
      "url": "https://palewi.re/docs/news-homepages/openai-gptbot-robotstxt.html",
      "type": "dataset",
      "publisher": "Ben Welsh / homepages.news",
      "verified": "2026-08-20",
      "keyFindings": "Open-source archive fetching robots.txt twice daily from 1,152 news publishers worldwide. Blocking rates: OpenAI 47.2% (544 sites), Common Crawl 47.0% (542), Google AI 42.4% (489). 593 of 1,152 publishers (51.5%) block at least one of the three. Sample spans national papers (NYT, WaPo) through small regional and non-English outlets. Longest-running reproducible time series on AI-bot blocking."
    },
    {
      "id": "pebblous-blocking-citation-gap",
      "title": "The Paradox of Blocking AI Crawlers: You Lose Visitors, Not Citations",
      "url": "https://blog.pebblous.ai/report/ai-crawler-blocking-citation-gap/en/",
      "type": "study",
      "publisher": "Pebblous, synthesizing BuzzStream/XOFU and Zhao (Rutgers) & Berman (Wharton)",
      "verified": "2026-08-20",
      "keyFindings": "BuzzStream/XOFU analysis of 4M AI citations across 3,600 prompts: sites that blocked AI crawlers retained citation presence at Google-Extended 92.3%, GPTBot 88.2%, OAI-SearchBot 82.4%, ChatGPT-User 70.6%; 'Even the sites that blocked training bots still accounted for about 95% of ChatGPT's citation sources.' Zhao & Berman traffic study: large publishers that blocked crawlers lost 23.1% of total traffic monthly (v1, Dec 2025), revised to roughly -7% weekly (v4, Apr 2026). Net finding: 'visitors are lost while citations remain' — blocking is a weak lever on training/citation but a real lever on referral traffic."
    },
    {
      "id": "peec",
      "title": "Peec AI",
      "url": "https://peec.ai/",
      "type": "vendor-doc",
      "publisher": "Peec AI",
      "verified": "2026-08-20",
      "keyFindings": "Exclusively prompt-rank monitoring: Prompt Management, per-model trackers (ChatGPT/Perplexity/Gemini), competitor benchmarking, Visibility/Position/Sentiment metrics, geo tracking, source detection, CSV export, Looker Studio connector, API. No technical site audit of any kind."
    },
    {
      "id": "perplexity-bots",
      "title": "PerplexityBot and Perplexity-User",
      "url": "https://docs.perplexity.ai/guides/bots",
      "publisher": "Perplexity",
      "type": "vendor-doc",
      "keyFindings": "Documents PerplexityBot (search indexing, not model training) and Perplexity-User (user-triggered fetches that generally ignore robots.txt). Addresses only user-agent identification, IP whitelisting and WAF configuration — no response headers, security headers, security.txt, trust scoring, page speed or JS rendering.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "perplexity-bots-docs",
      "title": "Perplexity Crawlers",
      "url": "https://docs.perplexity.ai/guides/bots",
      "type": "vendor-doc",
      "publisher": "Perplexity",
      "verified": "2026-08-21",
      "keyFindings": "Documents PerplexityBot ('designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models') and Perplexity-User ('supports user actions within Perplexity. When users ask Perplexity a question, it might visit a web page to help provide an accurate answer'). Scope is limited to crawler identification, IP ranges, robots.txt and WAF configuration. Notable ABSENCE: Perplexity publishes no guidance on page content structure, headings, direct answers, FAQ blocks, tables or dates for citation eligibility. Perplexity's help-center articles returned HTTP 403 to automated fetch and could not be independently verified."
    },
    {
      "id": "perplexity-crawlers-doc",
      "title": "Perplexity Crawlers",
      "url": "https://docs.perplexity.ai/docs/resources/perplexity-crawlers",
      "type": "vendor-doc",
      "publisher": "Perplexity",
      "verified": "2026-08-20",
      "keyFindings": "COUNTER-EVIDENCE. Perplexity's crawler doc names robots.txt as the sole webmaster control mechanism and never states that PerplexityBot consumes llms.txt. The only llms.txt reference is Perplexity publishing its own docs index. Publishing != consuming."
    },
    {
      "id": "perplexity-crawlers-docs",
      "title": "Perplexity Crawlers",
      "url": "https://docs.perplexity.ai/docs/resources/perplexity-crawlers",
      "type": "vendor-doc",
      "publisher": "Perplexity",
      "verified": "2026-08-20",
      "keyFindings": "PerplexityBot (search indexing, 'not used to crawl content for AI foundation models') and Perplexity-User (user-triggered). Direct quote: 'Since a user requested the fetch, this fetcher generally ignores robots.txt rules.' IP lists at perplexity.com/perplexitybot.json and /perplexity-user.json. No statements on JS rendering, sitemaps, caching, HTTPS or content types."
    },
    {
      "id": "perplexity-merchant-program-terms",
      "title": "Perplexity Merchant Program Terms of Service",
      "url": "https://www.perplexity.ai/hub/legal/merchant-program-terms-of-service",
      "type": "vendor-doc",
      "publisher": "Perplexity",
      "verified": "2026-08-20",
      "keyFindings": "Reported to define Product Data (title, price, description, brand, model, availability, shipping, dimensions, images, warranties, returns) and to require delivery \"as CSV or XML files in one of the formats specified by Perplexity\" — i.e. feed-based, not markup-based. Perplexity's hub returns HTTP 403 to WebFetch, so this could not be verified first-hand; treat as unverified and do not cite as proof."
    },
    {
      "id": "perplexity-source-labels",
      "title": "Understanding source labels",
      "url": "https://www.perplexity.ai/help-center/en/articles/20260806-understanding-source-labels",
      "type": "vendor-doc",
      "publisher": "Perplexity AI",
      "verified": "2026-08-20",
      "keyFindings": "Perplexity's help-center article on how it labels sources (reportedly rating whole websites rather than individual pages, using objective review questions, with partnerships stated not to affect labels). Fetch returned HTTP 403 so contents could not be verified first-hand; listed for completeness and NOT relied upon for any grade in this dossier."
    },
    {
      "id": "pew-ai-summaries-clicks",
      "title": "Google users are less likely to click on links when an AI summary appears in the results",
      "url": "https://www.pewresearch.org/short-reads/2025/07/22/google-users-are-less-likely-to-click-on-links-when-an-ai-summary-appears-in-the-results/",
      "type": "study",
      "publisher": "Pew Research Center",
      "verified": "2026-08-20",
      "keyFindings": "Published 2025-07-22. Methodology: 900 U.S. adults from KnowledgePanel Digital sharing real browsing data, March 2025; 68,879 unique Google searches, of which 12,593 produced an AI summary; results captured April 7–17, 2025. With an AI summary present users clicked a traditional result on 8% of visits vs 15% without (roughly half). Clicks on links inside the AI summary itself: 1% of visits. Users ended their browsing session on 26% of AI-summary visits vs 16% without. Best-methodology public evidence on AI answer cannibalization of clicks."
    },
    {
      "id": "playwright-aria-snapshots",
      "title": "Aria snapshots",
      "url": "https://playwright.dev/docs/aria-snapshots",
      "publisher": "Microsoft / Playwright",
      "type": "vendor-doc",
      "keyFindings": "Aria snapshots capture the accessibility tree as YAML describing roles, attributes, values and text content, 'derived from ARIA attributes or calculated based on HTML semantics'. Confirms the representation class agents consume; makes no mention of skip links.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "playwright-mcp-repo",
      "title": "microsoft/playwright-mcp README",
      "url": "https://github.com/microsoft/playwright-mcp",
      "type": "vendor-doc",
      "publisher": "Microsoft",
      "verified": "2026-08-21",
      "keyFindings": "'Uses Playwright's accessibility tree, not pixel-based input.' Tools: browser_snapshot (accessibility snapshot), browser_find (search the accessibility snapshot), browser_click / browser_type / browser_fill_form / browser_select_option all act on snapshot refs. browser_take_screenshot is explicitly 'for reference only, not for actions'."
    },
    {
      "id": "playwright-mcp-snapshots",
      "title": "Snapshots — Playwright MCP",
      "url": "https://playwright.dev/mcp/snapshots",
      "type": "vendor-doc",
      "publisher": "Microsoft",
      "verified": "2026-08-20",
      "keyFindings": "'Playwright MCP uses accessibility snapshots instead of screenshots.' 'Every tool that interacts with the page returns a structured tree of accessible elements with refs for interaction.' Snapshot contents explicitly include roles and landmarks, headings with levels, lists, contentinfo sections, form elements (textboxes, checkboxes) with accessible names, links and buttons, and per-element refs (e1, e15). Token comparison: ~200-400 tokens for a snapshot vs ~3,000-5,000 for a vision screenshot. Vision mode is the documented fallback for 'pages with poor accessibility markup'."
    },
    {
      "id": "playwright-permissions",
      "title": "BrowserContext.grantPermissions()",
      "url": "https://playwright.dev/docs/api/class-browsercontext#browser-context-grant-permissions",
      "publisher": "Microsoft / Playwright",
      "type": "vendor-doc",
      "keyFindings": "Permissions must be explicitly granted per context; they are not granted by default and no permission prompt is surfaced to automated browsers. Automation harnesses are structurally immune to the modal-prompt failure the audit describes.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "probe-aria-snapshot",
      "title": "Live Chromium accessibility snapshot of a probe page (own experiment)",
      "url": "https://playwright.dev/docs/aria-snapshots",
      "publisher": "Own experiment via Playwright MCP + Chromium",
      "type": "study",
      "keyFindings": "On a probe page with a canonical off-screen skip link, the snapshot rendered it as a plain extra node: `link \"Skip to main content\"` with `/url: \"#main-content\"`, immediately followed by `navigation` and `main [ref=e6]`. The skip link added a token-costing node and conferred no jump capability; the `main` landmark already bounded primary content.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "probe-aria-snapshot-address",
      "title": "Live Chromium accessibility snapshot of a probe page (own experiment)",
      "url": "https://playwright.dev/docs/aria-snapshots",
      "publisher": "Own experiment via Playwright MCP + Chromium",
      "type": "study",
      "keyFindings": "A footer <address> containing a mailto link and a street address rendered as an unnamed `generic [ref=e17]` node — identical in shape to a plain div — while an equivalent contact block in a <p> rendered as `paragraph [ref=e19]`. The agent-facing accessibility tree carries no 'contact information' semantics for <address>.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "probe-aria-snapshot-aside",
      "title": "Live Chromium accessibility snapshot of a probe page (own experiment)",
      "url": "https://playwright.dev/docs/aria-snapshots",
      "publisher": "Own experiment via Playwright MCP + Chromium",
      "type": "study",
      "keyFindings": "Both a top-level and a nested <aside> surfaced in the agent-facing snapshot as `complementary [ref=e10]` and `complementary [ref=e14]`, nested inside `main [ref=e6]`. Confirms an agent reading the a11y tree can distinguish supplementary blocks by role; the same content in a bare <div> would have appeared as an undifferentiated `generic` node.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "probe-aria-snapshot-images",
      "title": "Live Chromium accessibility snapshot of a probe page (own experiment)",
      "url": "https://playwright.dev/docs/aria-snapshots",
      "publisher": "Own experiment via Playwright MCP + Chromium",
      "type": "study",
      "keyFindings": "Three same-source images: `alt=\"\"` (no role), `alt=\"\" role=\"presentation\"`, and `alt=\"A described chart\"`. The snapshot contained only `img \"A described chart\" [ref=e9]`. Both empty-alt images were omitted identically, with and without the role attribute — the attribute the audit requires changes nothing an agent sees, and empty alt removes the image from the agent's view entirely.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "probe-cloudflare-mcp-json",
      "title": "Live deployment: Cloudflare /.well-known/mcp.json",
      "url": "https://cloudflare.com/.well-known/mcp.json",
      "type": "dataset",
      "publisher": "Cloudflare (probed by Agent Lighthouse research, 2026-08-20)",
      "verified": "2026-08-20",
      "keyFindings": "HTTP 200 application/json. Format is a `{\"mcpServers\":{...}}` map (mirrors Claude-desktop client config), with transport.url pointing at https://www.cloudflare.com/.well-known/webmcp.json and docs at /llms.txt. Structurally INCOMPATIBLE with Notion's document at the same path."
    },
    {
      "id": "probe-notion-mcp-json",
      "title": "Live deployment: Notion /.well-known/mcp.json",
      "url": "https://notion.so/.well-known/mcp.json",
      "type": "dataset",
      "publisher": "Notion (probed by Agent Lighthouse research, 2026-08-20)",
      "verified": "2026-08-20",
      "keyFindings": "HTTP 200. Body: {\"name\":\"Notion\",\"description\":...,\"icon\":...,\"endpoint\":\"https://mcp.notion.com/mcp\"} — a flat single-server object. Demonstrates that `/.well-known/mcp.json` has at least three mutually incompatible in-the-wild shapes (Notion flat, Cloudflare mcpServers-map, and the various SEP drafts)."
    },
    {
      "id": "probe-vercel-ai-catalog",
      "title": "Live deployment: Vercel /.well-known/ai-catalog.json",
      "url": "https://vercel.com/.well-known/ai-catalog.json",
      "type": "dataset",
      "publisher": "Vercel (probed by Agent Lighthouse research, 2026-08-20)",
      "verified": "2026-08-20",
      "keyFindings": "HTTP 200 with the spec-correct media type `application/ai-catalog+json`. Body: {\"specVersion\":\"1.0\",\"host\":{\"displayName\":\"Vercel\",...},\"entries\":[...]} with entries using `urn:air:vercel.com:...` identifiers and linking mcp.vercel.com/.well-known/oauth-protected-resource, https://vercel.com/openapi.json, and https://vercel.com/docs/agent-resources.md. This is a conformant production implementation of a spec that is still an open, unmerged SEP."
    },
    {
      "id": "probe-vercel-api-catalog",
      "title": "Live deployment: Vercel /.well-known/api-catalog (RFC 9727)",
      "url": "https://vercel.com/.well-known/api-catalog",
      "type": "dataset",
      "publisher": "Vercel (probed by Agent Lighthouse research, 2026-08-20)",
      "verified": "2026-08-20",
      "keyFindings": "HTTP 200 with `application/linkset+json; profile=\"https://www.rfc-editor.org/info/rfc9727\"` — a fully conformant RFC 9727 response including the profile parameter. Linkset anchors https://api.vercel.com with service-desc → https://openapi.vercel.sh/, service-doc, and status links. A fifth conformant provider beyond the four API Evangelist found in May 2026."
    },
    {
      "id": "probe-zapier-ai-catalog",
      "title": "Live deployment: Zapier /.well-known/ai-catalog.json",
      "url": "https://zapier.com/.well-known/ai-catalog.json",
      "type": "dataset",
      "publisher": "Zapier (probed by Agent Lighthouse research, 2026-08-20)",
      "verified": "2026-08-20",
      "keyFindings": "HTTP 200 application/json. specVersion 1.0, host.trustManifest present, entry `urn:air:zapier.com:mcp` with type `application/mcp-server-card+json` pointing at https://zapier.com/.well-known/mcp/server-card.json, plus `representativeQueries` — a field not in the MCP extension's documented entry table (vendor extension)."
    },
    {
      "id": "probe-zapier-api-catalog",
      "title": "Live deployment: Zapier /.well-known/api-catalog",
      "url": "https://zapier.com/.well-known/api-catalog",
      "type": "dataset",
      "publisher": "Zapier (probed by Agent Lighthouse research, 2026-08-20)",
      "verified": "2026-08-20",
      "keyFindings": "HTTP 200 application/linkset+json (no profile parameter — RFC 9727 says SHOULD). Anchors https://mcp.zapier.com/mcp with service-desc → the MCP server card, showing RFC 9727 being repurposed as an MCP discovery carrier."
    },
    {
      "id": "probe-zapier-server-card",
      "title": "Live deployment: Zapier /.well-known/mcp/server-card.json",
      "url": "https://zapier.com/.well-known/mcp/server-card.json",
      "type": "dataset",
      "publisher": "Zapier (probed by Agent Lighthouse research, 2026-08-20)",
      "verified": "2026-08-20",
      "keyFindings": "HTTP 200 application/json. Card: name 'com.zapier/mcp', version 1.0.0, remotes[{type:'streamable-http', url:'https://mcp.zapier.com/api/v1/connect', supportedProtocolVersions:['2025-11-25','2025-06-18','2025-03-26','2024-11-05','2024-10-07']}], capabilities.tools, auth oauth2. Note this uses the SUPERSEDED SEP-1649 path, not SEP-2127's recommended `<streamable-http-url>/server-card`."
    },
    {
      "id": "probe-zapier-webmcp-manifest",
      "title": "Live deployment: Zapier /.well-known/webmcp",
      "url": "https://zapier.com/.well-known/webmcp",
      "type": "dataset",
      "publisher": "Zapier (probed by Agent Lighthouse research, 2026-08-20)",
      "verified": "2026-08-20",
      "keyFindings": "HTTP 200 application/json with a self-declared, vendor-invented schema: {\"spec\":\"zapier-webmcp-discovery/1\",...}. Its own description concedes the mechanism: 'Pre-navigation discovery map... These tools are registered client-side on the pages below via the browser's WebMCP API (document.modelContext) and are invoked in-page by a browser-resident agent — they are not HTTP endpoints.' Confirms there is no standard manifest — vendors are inventing their own."
    },
    {
      "id": "profound",
      "title": "Profound — AI visibility platform",
      "url": "https://www.tryprofound.com/",
      "type": "vendor-doc",
      "publisher": "Profound",
      "verified": "2026-08-20",
      "keyFindings": "Shipped features: Answer Engine Insights, Prompt Volumes, Agent Analytics (crawl frequency by bot, from the customer's own logs/edge), Shopping, Agents (content generation), Aim (task prioritisation). Zero active site-side audits: no robots.txt parsing, no llms.txt validation, no schema checks, no differential fetching."
    },
    {
      "id": "profound-homepage",
      "title": "Profound — AI search visibility platform",
      "url": "https://www.tryprofound.com/",
      "type": "vendor-doc",
      "publisher": "Profound",
      "verified": "2026-08-20",
      "keyFindings": "Products: Answer Engine Insights, Agent Analytics ('interpreted and crawled by ChatGPT, Gemini, Claude, Perplexity'), Prompt Volumes, plus a free AEO Report scoring AI Visibility, Source Citations, Brand Sentiment and Content AEO. The homepage does not describe data collection methodology, bot-log methodology, prompt-sampling design, or scoring weights, and cites no external evidence for any check."
    },
    {
      "id": "pulsemcp-servers",
      "title": "PulseMCP — MCP server directory",
      "url": "https://www.pulsemcp.com/servers",
      "type": "dataset",
      "publisher": "PulseMCP",
      "verified": "2026-08-20",
      "keyFindings": "Directory lists 22,033 MCP servers, described as daily-updated with a visible last-update timestamp. Note at time of fetch: new server submissions and listing changes paused until mid-August pending an ingestion rework, so the count may lag. Aggregates third-party and community servers; no dedupe methodology published."
    },
    {
      "id": "rankability-adoption-tracker",
      "title": "LLMS.txt Adoption Tracker (Tranco top 1,000)",
      "url": "https://www.rankability.com/data/llms-txt-adoption/",
      "type": "dataset",
      "publisher": "Rankability",
      "verified": "2026-08-20",
      "keyFindings": "Best adoption dataset. June 2026, Tranco top 1,000 domains, HTTPS fetch with an identified crawler, counted as adopters only on HTTP 200 with real plain-text content. RESULTS: 8.7% of top 1,000 publish llms.txt (87 sites); 15.8% of the 549 reachable sites (451 were unreachable infrastructure domains). llms-full.txt: only 15 sites (1.5%); 15 publish both; 72 publish llms.txt only; ZERO publish llms-full.txt alone. Tracks no format-quality or crawler-consumption metrics."
    },
    {
      "id": "readability-repo",
      "title": "mozilla/readability",
      "url": "https://github.com/mozilla/readability",
      "type": "repo",
      "publisher": "Mozilla",
      "verified": "2026-08-20",
      "keyFindings": "parse() returns title, content, textContent, length, excerpt, byline, dir, siteName, lang, publishedTime; charThreshold default 500 chars below which no article is returned; isProbablyReaderable uses minContentLength 140 and minScore 20. Gives concrete pass/fail hooks (null result, length, title) for an extractability check."
    },
    {
      "id": "readability-src",
      "title": "Readability.js source — _isProbablyVisible",
      "url": "https://raw.githubusercontent.com/mozilla/readability/main/Readability.js",
      "type": "repo",
      "publisher": "Mozilla",
      "verified": "2026-08-20",
      "keyFindings": "Visibility test is literally: node.style.display != \"none\" && node.style.visibility != \"hidden\" && !node.hasAttribute(\"hidden\") && aria-hidden!=\"true\". Only inline styles and attributes are consulted — \"It does not evaluate class-based CSS rules from stylesheets.\" Proof that content hidden by an external stylesheet class is ingested as if visible by the most widely deployed extractor."
    },
    {
      "id": "reuters-institute-dnr-2025",
      "title": "Reuters Institute Digital News Report 2025 — Executive Summary",
      "url": "https://reutersinstitute.politics.ox.ac.uk/digital-news-report/2025/dnr-executive-summary",
      "type": "study",
      "publisher": "Reuters Institute for the Study of Journalism, University of Oxford",
      "verified": "2026-08-20",
      "keyFindings": "Survey across six continents and 48 markets (Serbia added). Weekly use of AI chatbots for news is 7% globally, ~15% among under-25s; India 18%, UK 3%. Overall news trust flat at 40% for a third year. Trusted news brands remain the most-named destination for fact-checking. Caveat stated in the report: some markets rely on educated, English-speaking online samples, limiting cross-market comparability."
    },
    {
      "id": "rfc-3966-tel-uri",
      "title": "RFC 3966 — The tel URI for Telephone Numbers",
      "url": "https://www.rfc-editor.org/rfc/rfc3966.html",
      "type": "spec",
      "publisher": "IETF",
      "verified": "2026-08-20",
      "keyFindings": "Ratified Standards Track definition of the tel: URI scheme. Consumers are browsers/OS dialers; no AI-agent vendor documents tel: extraction as a task-completion affordance."
    },
    {
      "id": "rfc-4287-atom",
      "title": "RFC 4287 — The Atom Syndication Format",
      "url": "https://www.rfc-editor.org/rfc/rfc4287.html",
      "type": "spec",
      "publisher": "IETF",
      "verified": "2026-08-21",
      "keyFindings": "Standards Track, December 2005. 'This document specifies Atom, an XML-based Web content and metadata syndication format.' Ratified standard; note it does NOT itself define HTML <link rel=alternate> feed autodiscovery."
    },
    {
      "id": "rfc-6596-canonical",
      "title": "RFC 6596 — The Canonical Link Relation",
      "url": "https://www.rfc-editor.org/rfc/rfc6596.html",
      "type": "spec",
      "publisher": "IETF",
      "verified": "2026-08-21",
      "keyFindings": "Informational RFC (not Standards Track). Defines 'canonical' as specifying 'the preferred IRI from a set of resources that return the context IRI's content in duplicated form'. Names search engines as the intended consumer class: applications 'can' index only the canonical version, consolidate link popularity metrics, and display the target URL as representative."
    },
    {
      "id": "rfc-8615",
      "title": "RFC 8615 — Well-Known Uniform Resource Identifiers (URIs)",
      "url": "https://www.rfc-editor.org/rfc/rfc8615.html",
      "type": "spec",
      "publisher": "IETF",
      "keyFindings": "Defines the `/.well-known/` path prefix that `api-catalog`, `ai-catalog.json` and `agent-card.json` all register under.",
      "verified": "2026-08-20"
    },
    {
      "id": "rfc-9116-security-txt",
      "title": "RFC 9116 — A File Format to Aid in Security Vulnerability Disclosure",
      "url": "https://www.rfc-editor.org/rfc/rfc9116.html",
      "type": "spec",
      "publisher": "IETF",
      "verified": "2026-08-20",
      "keyFindings": "INFORMATIONAL RFC, explicitly 'not an Internet Standards Track specification'. File lives at https://example.com/.well-known/security.txt, served as text/plain, UTF-8, over HTTPS only. Required fields: Contact and Expires. Intended consumers are security researchers; the RFC cautions that 'researchers must review all information within the security.txt file before submitting reports in an automated fashion'. No AI agent is named as a consumer."
    },
    {
      "id": "rfc-9309",
      "title": "RFC 9309: Robots Exclusion Protocol",
      "url": "https://www.rfc-editor.org/rfc/rfc9309.html",
      "type": "spec",
      "publisher": "IETF",
      "verified": "2026-08-21",
      "keyFindings": "Ratified Sept 2022. §2.2.1 product token matching MUST be case-insensitive; multiple matching groups MUST be combined into one group; fall back to '*' group; if no group matches and no '*', no rules apply. §2.2.2 matching SHOULD be case-sensitive on paths, MUST start at first octet, 'most specific match = most octets' wins, allow beats disallow on ties, no match = allowed, /robots.txt implicitly allowed, percent-encoding normalization rules, rules outside any group SHOULD be ignored. §2.2.3 mandates support for '*' (0+ chars), '$' (end of match), '#' (comment). §2.3.1.3 4xx = crawler MAY access anything; §2.3.1.4 5xx/network error = crawler MUST assume complete disallow. §2.4 cache SHOULD NOT exceed 24h. §2.5 parsing limit MUST be at least 500 KiB. §2.2.4 crawlers MAY interpret non-protocol records such as Sitemaps. CRITICAL NEGATIVE: the RFC text contains NO occurrence of 'BOM' or 'byte order mark', and NO crawl-delay directive."
    },
    {
      "id": "rfc-9309-txt",
      "title": "RFC 9309 plain-text canonical",
      "url": "https://www.rfc-editor.org/rfc/rfc9309.txt",
      "type": "spec",
      "publisher": "IETF",
      "verified": "2026-08-20",
      "keyFindings": "Full-text search confirms zero matches for 'BOM' / 'byte order mark'. Confirms verbatim §2.2.1, §2.2.2, §2.2.3, §2.3.1, §2.5 text quoted above."
    },
    {
      "id": "rfc-9727",
      "title": "RFC 9727 — api-catalog: A Well-Known URI and Link Relation to Help Discovery of APIs",
      "url": "https://www.rfc-editor.org/rfc/rfc9727.html",
      "type": "spec",
      "publisher": "IETF",
      "verified": "2026-08-21",
      "keyFindings": "Standards Track, published June 2025, author K. Smith (Vodafone). Registers `/.well-known/api-catalog` and the `api-catalog` link relation. 'A supporting publisher SHALL resolve an HTTPS GET request to /.well-known/api-catalog and return an API catalog document.' Catalog MUST be served as Linkset (`application/linkset+json`) and SHOULD carry `profile=\"https://www.rfc-editor.org/info/rfc9727\"`."
    },
    {
      "id": "rfc4287",
      "title": "RFC 4287 — The Atom Syndication Format",
      "url": "https://www.rfc-editor.org/rfc/rfc4287",
      "type": "spec",
      "publisher": "IETF",
      "verified": "2026-08-20",
      "keyFindings": "Sec 4.1.2: atom:entry MUST contain exactly one atom:id (permanent, universally unique IRI that 'must not change across different instantiations of the entry') and exactly one atom:updated ('most recent modification time that the publisher considers significant'). atom:entry MUST contain atom:summary when atom:content carries a src attribute (and is thus empty), or when content is Base64-encoded. MUST NOT contain more than one atom:summary."
    },
    {
      "id": "rfc7763",
      "title": "RFC 7763 — The text/markdown Media Type",
      "url": "https://www.rfc-editor.org/rfc/rfc7763.html",
      "type": "spec",
      "publisher": "IETF",
      "verified": "2026-08-20",
      "keyFindings": "Registers type text/markdown with required charset parameter and optional variant parameter (hint to the recipient, IANA registry of variants). Gives the markdown-alternate check a ratified Content-Type to assert against instead of accepting text/plain or an HTML fallback."
    },
    {
      "id": "rfc9110-safe",
      "title": "RFC 9110 §9.2.1 — Safe Methods",
      "url": "https://www.rfc-editor.org/rfc/rfc9110.html#name-safe-methods",
      "type": "spec",
      "publisher": "IETF",
      "verified": "2026-08-20",
      "keyFindings": "'Request methods are considered safe if their defined semantics are essentially read-only; they do not commit to any action on the origin server.' §3.5 notes spiders are configured to follow links while crawling the web as a hypertext graph. Ratified basis for the claim that an exploring agent may follow any GET link and expects no side effect."
    },
    {
      "id": "rfc9116",
      "title": "RFC 9116: A File Format to Aid in Security Vulnerability Disclosure",
      "url": "https://www.rfc-editor.org/rfc/rfc9116.html",
      "publisher": "IETF",
      "type": "spec",
      "keyFindings": "Abstract scopes the format to helping \"researchers to report vulnerabilities\". Section 1.1 frames the problem as researchers lacking reporting channels. The RFC does not mention search engines, crawlers, indexing services, or automated agents beyond security scanners (Section 5.8). Named consumer set excludes AI systems entirely.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "rfc9309",
      "title": "RFC 9309 — Robots Exclusion Protocol",
      "url": "https://www.rfc-editor.org/rfc/rfc9309.html",
      "type": "spec",
      "publisher": "IETF",
      "verified": "2026-08-21",
      "keyFindings": "§2.2.1: 'Crawlers MUST use case-insensitive matching to find the group that matches the product token and then obey the rules of the group.' Groups matching the SAME token are combined. Critically: 'If no matching group exists, crawlers MUST obey the group with a user-agent line with the \"*\" value, if present.' The wildcard group is a fallback only — it is never merged with a named group. A named AI-bot group therefore fully shadows every wildcard rule."
    },
    {
      "id": "rss-2-specification",
      "title": "RSS 2.0 Specification",
      "url": "https://www.rssboard.org/rss-specification",
      "type": "spec",
      "publisher": "RSS Advisory Board",
      "verified": "2026-08-21",
      "keyFindings": "Acknowledges full-content usage explicitly: \"An item may also be complete in itself, if so, the description contains the text\"."
    },
    {
      "id": "rss-autodiscovery-spec",
      "title": "RSS Autodiscovery",
      "url": "https://www.rssboard.org/rss-autodiscovery",
      "type": "spec",
      "publisher": "RSS Advisory Board",
      "verified": "2026-08-20",
      "keyFindings": "Specifies <link rel=\"alternate\" type=\"application/rss+xml\" href=\"...\"> in the head. Deployed since 2002 and near-universal in CMS output. Named consumers are browsers and feed aggregators. No AI system is named as a consumer."
    },
    {
      "id": "rss-content-module",
      "title": "RSS 1.0 content module",
      "url": "https://web.resource.org/rss/1.0/modules/content",
      "type": "spec",
      "publisher": "RSS-DEV Working Group",
      "verified": "2026-08-21",
      "keyFindings": "Defines `content:encoded` as \"An element whose contents are the entity-encoded or CDATA-escaped version of the content of the item\" — the item's full body carried inside the feed."
    },
    {
      "id": "s1",
      "title": "HTTP Message Signatures for Automated Traffic Protocol (draft-meunier-webbotauth-httpsig-protocol-02)",
      "url": "https://datatracker.ietf.org/doc/draft-meunier-webbotauth-httpsig-protocol/",
      "type": "draft-spec",
      "publisher": "IETF / Thibault Meunier (Cloudflare), Sandor Major (Google)",
      "verified": "2026-08-20",
      "keyFindings": "ACTIVE draft-02, last updated 2026-08-18. Defines three request headers: Signature, Signature-Input (params: created, expires, keyid, tag), and Signature-Agent (Structured Dictionary of HTTPS URLs, default type=\"directory\"). Defines well-known URI /.well-known/http-message-signatures-directory serving a JWKS with media type application/http-message-signatures-directory+json. Origin MUST parse the three headers, resolve Signature-Agent, validate; MAY return 400 Bad Request on malformed headers and 403 Forbidden when additional signatures are required."
    },
    {
      "id": "s10",
      "title": "A Vocabulary For Expressing AI Usage Preferences (draft-ietf-aipref-vocab-07)",
      "url": "https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/",
      "type": "draft-spec",
      "publisher": "IETF aipref WG (Paul Keller, Open Future; Martin Thomson, Mozilla)",
      "verified": "2026-08-20",
      "keyFindings": "ACTIVE, version 07, 2026-08-19, intended status Proposed Standard, WG-adopted. Categories: `train-ai` (modify learned parameters of a generative model) and `search` (select assets and direct users to their location, with excerpt conditions). Values are single-character tokens: `y` = allow, `n` = disallow, absent = unknown. Expressed as an RFC 8941 Structured Field dictionary, e.g. `train-ai=y, search=n`. Three-value outcome model: allowed / disallowed / unknown."
    },
    {
      "id": "s11",
      "title": "Attaching AI Usage Preferences to Content (draft-ietf-aipref-attach-05)",
      "url": "https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/",
      "type": "draft-spec",
      "publisher": "IETF aipref WG",
      "verified": "2026-08-20",
      "keyFindings": "ACTIVE, version 05, 2026-08-18. Two attachment mechanisms: (1) HTTP response header `Content-Usage: train-ai=n`; (2) robots.txt directive `Content-Usage: train-ai=n` with optional path prefix, e.g. `Content-Usage: /ai-ok/ train-ai=y`, using the same path-prefix matching as Allow/Disallow. Precedence rules: preferences apply ONLY to crawlable resources — \"Disallowed paths have no associated usage preferences\"; longest matching path prefix wins. No well-known location is defined."
    },
    {
      "id": "s12",
      "title": "RSL 1.0 Standard Specification",
      "url": "https://rslstandard.org/rsl",
      "type": "spec",
      "publisher": "RSL Collective",
      "verified": "2026-08-20",
      "keyFindings": "robots.txt directive `License: https://example.com/license.xml` — \"The value MUST be an absolute URI\"; may be global or inside a User-agent group; multiple allowed. HTTP discovery: `Link: <https://example.com/license.xml>; rel=\"license\"; type=\"application/rsl+xml\"`. HTML: `<link rel=\"license\" type=\"application/rsl+xml\" href=\"...\">` or inline `<script type=\"application/rsl+xml\">`. NO default/well-known location is mandated. XML: root `<rsl xmlns=\"https://rslstandard.org/rsl\" max-age>`, `<content url required, server, encrypted>`, `<license>`, `<permits|prohibits type=\"usage|user|geo\">`, `<payment type=\"purchase|subscription|crawl|use|attribution|free\">`, `<amount currency=ISO4217>`, `<standard>`, `<copyright type contactEmail contactUrl>`, `<legal type=\"warranty|disclaimer|attestation|contact|proof\">`."
    },
    {
      "id": "s13",
      "title": "Really Simple Licensing (RSL) — home",
      "url": "https://rslstandard.org/",
      "type": "spec",
      "publisher": "RSL Collective",
      "verified": "2026-08-20",
      "keyFindings": "RSL 1.0 released; open standard for machine-readable licensing incl. attribution, pay per crawl, pay per inference. Supporters listed: Akamai, Cloudflare, Creative Commons, Fastly, Reddit, O'Reilly Media, Vox Media, Yahoo, Ziff Davis. Discovery via rsl.xml files, robots.txt, HTTP headers, HTML, RSS, media files, plus an Encrypted Media Standard."
    },
    {
      "id": "s14",
      "title": "Detect a Challenge Page response (cf-mitigated)",
      "url": "https://developers.cloudflare.com/cloudflare-challenges/challenge-types/challenge-pages/detect-response/",
      "type": "vendor-doc",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "Exact header is `cf-mitigated`; \"challenge is the only valid value\"; \"The header is set for all Challenge Page types.\" Also: \"regardless of the requested resource-type, the content-type of a challenge will be text/html\". Gives a deterministic, vendor-documented way for an auditor to distinguish \"you were challenged\" from \"you were served content\" or \"you hit an ordinary 403\"."
    },
    {
      "id": "s15",
      "title": "Paywalled content structured data",
      "url": "https://developers.google.com/search/docs/appearance/structured-data/paywalled-content",
      "type": "vendor-doc",
      "publisher": "Google",
      "verified": "2026-08-20",
      "keyFindings": "`isAccessibleForFree: false` plus `hasPart` with `@type: WebPageElement`, `isAccessibleForFree: false` and a `cssSelector` naming the restricted region. Applies to CreativeWork subtypes (Article, NewsArticle, Blog, Course, HowTo, Review, WebPage, Comment, Message). Google states this markup \"helps Google differentiate paywalled content from the practice of cloaking, which violates spam policies\" — i.e. serving less content to a crawler is only safe when declared."
    },
    {
      "id": "s16",
      "title": "RFC 9421: HTTP Message Signatures",
      "url": "https://www.rfc-editor.org/rfc/rfc9421.html",
      "type": "spec",
      "publisher": "IETF",
      "verified": "2026-08-20",
      "keyFindings": "Ratified standard underlying Web Bot Auth. §4.1 Signature-Input, §4.2 Signature. §2.3 defines the `tag` parameter: \"An application-specific tag for the signature as a String value... used by applications to help identify signatures relevant for specific applications or protocols\" (web-bot-auth uses tag=\"web-bot-auth\"). §5.1 defines the `Accept-Signature` response field for negotiating signatures in subsequent messages — the only standardised way an origin can advertise that it wants signed requests."
    },
    {
      "id": "s17",
      "title": "TDM Reservation Protocol (TDMRep) — W3C CG Final Report",
      "url": "https://www.w3.org/community/reports/tdmrep/CG-FINAL-tdmrep-20240202/",
      "type": "spec",
      "publisher": "W3C Community Group",
      "verified": "2026-08-20",
      "keyFindings": "Four techniques, not three: (1) `/.well-known/tdmrep.json` — an ARRAY of objects each with `location`, `tdm-reservation`, `tdm-policy`; (2) HTTP response headers `tdm-reservation: 1` and `tdm-policy: <url>`; (3) HTML `<meta name=\"tdm-reservation\" content=\"1\">` / `<meta name=\"tdm-policy\" ...>`; (4) EPUB `tdm:reservation`/`tdm:policy`. Note the well-known file is an array of rules — a bare object is non-conformant."
    },
    {
      "id": "s18",
      "title": "OpenAI crawlers and user agents",
      "url": "https://developers.openai.com/api/docs/bots",
      "type": "vendor-doc",
      "publisher": "OpenAI",
      "verified": "2026-08-21",
      "keyFindings": "Exact UA strings and published IP-range JSONs. OAI-SearchBot (…compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot) → https://openai.com/searchbot.json. GPTBot (…compatible; GPTBot/1.4; +https://openai.com/gptbot) → https://openai.com/gptbot.json. ChatGPT-User (…compatible; ChatGPT-User/1.0; +https://openai.com/bot) → https://openai.com/chatgpt-user.json. OAI-AdsBot → https://openai.com/adsbot.json. All four JSON endpoints return HTTP 200 (curl-verified). No mention of Web Bot Auth. Note Google-Extended has no UA at all, so it cannot be probed by request."
    },
    {
      "id": "s19",
      "title": "x402 Specification v2",
      "url": "https://github.com/x402-foundation/x402/blob/main/specs/x402-specification-v2.md",
      "type": "spec",
      "publisher": "x402 Foundation",
      "verified": "2026-08-20",
      "keyFindings": "PaymentRequired payload carries `\"x402Version\": 2` and an `accepts` array whose items have `scheme`, `network`, `amount`, `asset`, `payTo`, `maxTimeoutSeconds`, `extra`. \"For HTTP, the canonical wire location is the base64-encoded `PAYMENT-REQUIRED` response header\"; a `PAYMENT-SIGNATURE` header appears in examples. Protocol is transport-agnostic; HTTP binding lives in specs/transports-v2/http.md. Note the older repo path coinbase/x402 and specs/x402-specification.md are now 404."
    },
    {
      "id": "s2",
      "title": "Web Bot Auth Architecture (draft-meunier-web-bot-auth-architecture-05)",
      "url": "https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/",
      "type": "draft-spec",
      "publisher": "IETF",
      "verified": "2026-08-20",
      "keyFindings": "Version 05, last updated 2026-03-02, now EXPIRED and replaced by draft-meunier-webbotauth-httpsig-protocol. Cite s1, not this, for current header semantics; useful only as the architectural rationale (identifying automated traffic via HTTP Message Signatures)."
    },
    {
      "id": "s20",
      "title": "Anubis (proof-of-work interstitial)",
      "url": "https://anubis.techaro.lol/docs/design/why-proof-of-work/",
      "type": "vendor-doc",
      "publisher": "Techaro",
      "verified": "2026-08-20",
      "keyFindings": "URL resolves but served an Anubis deny page rather than the doc — which itself confirmed the fingerprints an auditor needs: asset paths under `/.within.website/x/cmd/anubis/`, body text \"Access Denied: error code <hex>\", footer \"Protected by Anubis From Techaro\", and a version banner (v1.27.1-…). Treat as a fingerprint source, not as a cited spec; re-verify the doc URL before shipping it in remediation copy."
    },
    {
      "id": "s21",
      "title": "AI Crawl Control — Cloudflare",
      "url": "https://developers.cloudflare.com/ai-crawl-control/",
      "type": "vendor-doc",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "\"Works automatically on all Cloudflare plans\"; operators \"Set allow or block rules for individual crawlers\" and monitor AI access. Default-block posture and blocked-crawler response codes are NOT documented on this page — so an auditor must not assert Cloudflare's default behaviour, only measure the observed response."
    },
    {
      "id": "s3",
      "title": "Web Bot Auth — Cloudflare Bots docs",
      "url": "https://developers.cloudflare.com/bots/concepts/bot/verified-bots/web-bot-auth/",
      "type": "vendor-doc",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "Confirms the deployed shape of the three headers: Signature-Input carries tag=\"web-bot-auth\", keyid, created/expires, nonce, alg; Signature-Agent is a structured string such as \"https://signature-agent.test\". Bots must host a JWKS at /.well-known/http-message-signatures-directory over HTTPS. Cloudflare verifies server-side against its registered bot database; site operators configure nothing themselves — which is exactly why a site can silently reject signed traffic without knowing."
    },
    {
      "id": "s4",
      "title": "Verified bots policy — Cloudflare",
      "url": "https://developers.cloudflare.com/bots/concepts/bot/verified-bots/policy/",
      "type": "vendor-doc",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "Two requirements for verified status: (1) \"Honest self-identification — it declares who it is deterministically, through a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS.\" (2) \"Non-abusive behavior — it obeys robots.txt and crawl directives...\". Establishes that UA-string alone is never trusted, which is the source of the false-positive ambiguity when auditing edge blocks by UA spoofing."
    },
    {
      "id": "s5",
      "title": "Introducing pay per crawl: Enabling content owners to charge AI crawlers",
      "url": "https://blog.cloudflare.com/introducing-pay-per-crawl/",
      "type": "vendor-doc",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "Exact wire format for paid crawling. Server returns HTTP 402 Payment Required with `crawler-price: USD XX.XX`. Crawler retries with `crawler-exact-price: USD XX.XX`, or pre-declares with `crawler-max-price: USD XX.XX`. Successful paid fetch returns 200 with `crawler-charged: USD XX.XX`."
    },
    {
      "id": "s6",
      "title": "Pay per crawl — AI Crawl Control",
      "url": "https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/",
      "type": "vendor-doc",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "Feature landing page confirming pay-per-crawl ships as a product surface under AI Crawl Control (private beta). Technical detail lives in the blog post (s5); this is the canonical product doc URL to cite in remediation text."
    },
    {
      "id": "s7",
      "title": "Managed robots.txt — Cloudflare Bots",
      "url": "https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/",
      "type": "vendor-doc",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "Cloudflare PREPENDS its own block to the origin's robots.txt: `User-Agent: *` / `Content-signal: search=yes, ai-train=no, use=reference` / `Allow: /`. When the origin already serves robots.txt it combines both files, Cloudflare's block first. This is the mechanism by which a site's own AI policy gets silently overridden at the edge — directly auditable by diffing declared vs. served robots.txt."
    },
    {
      "id": "s8",
      "title": "Content Signals Policy (announcement)",
      "url": "https://blog.cloudflare.com/content-signals-policy/",
      "type": "vendor-doc",
      "publisher": "Cloudflare",
      "verified": "2026-08-20",
      "keyFindings": "Defines the `Content-Signal:` robots.txt directive with signals `search`, `ai-input`, `ai-train` and values `yes`/`no` (omission = no preference). Canonical example: `User-Agent: *` / `Content-Signal: search=yes, ai-train=no` / `Allow: /`. Deployed as default on ~3.8M Cloudflare domains. Cloudflare deliberately does not emit an `ai-input` signal."
    },
    {
      "id": "s9",
      "title": "Content Signals (AIPREF guide)",
      "url": "https://contentsignals.org/",
      "type": "article",
      "publisher": "Cloudflare / contentsignals.org",
      "verified": "2026-08-20",
      "keyFindings": "Resolves HTTP 200. Page now self-describes as \"An up-to-date guide to the IETF's proposed new AI Preferences (aipref)\" — i.e. the Content-Signal vocabulary is being folded into the IETF AIPREF work (s10/s11). Body is JS-rendered behind Cloudflare, so scrape the meta description or the drafts directly."
    },
    {
      "id": "sandboxseo-geo-critique",
      "title": "GEO Targeted: Critiquing the Generative Engine Optimization Research",
      "url": "https://sandboxseo.com/generative-engine-optimization-experiment/",
      "type": "article",
      "publisher": "Sandbox SEO (Tylor Hermanson)",
      "verified": "2026-08-20",
      "keyFindings": "Four methodological objections to the GEO paper: (1) confound — the three winning tactics (Quotation, Cite Sources, Statistics) all ADD content while the six losers only tweak existing text, so gains may reflect added information rather than the tactic; (2) fabrication — the optimizations permitted entirely LLM-invented quotes, statistics and sources, and 'fake information dreamed up by an LLM is that it's truly unique', so real-world results are unlikely to replicate because genuine quotes/stats are harder to make unique; (3) biased prompts — the LLM was told to produce content that is 'correct, high-quality', 'informative, interesting, and engaging', inherently favouring citation-rich fluent output; (4) restricted candidate set — testing only top-5 results creates artificial scarcity that exaggerates the 30–40% gains. No independent replication was attempted."
    },
    {
      "id": "schema-aggregateoffer",
      "title": "schema.org — AggregateOffer",
      "url": "https://schema.org/AggregateOffer",
      "type": "spec",
      "publisher": "schema.org",
      "verified": "2026-08-21",
      "keyFindings": "Defines `lowPrice`, `highPrice` and `offerCount` for the multi-offer case; no `price` is mandated."
    },
    {
      "id": "schema-aggregaterating",
      "title": "schema.org — AggregateRating",
      "url": "https://schema.org/AggregateRating",
      "type": "spec",
      "publisher": "schema.org",
      "verified": "2026-08-21",
      "keyFindings": "Core ratified vocabulary — \"The average rating based on multiple ratings or reviews\" — with `ratingValue`, `reviewCount` and `ratingCount`. Deployed on 1M–10M domains per the July 2026 aggregation of Google's web index."
    },
    {
      "id": "schema-breadcrumblist",
      "title": "BreadcrumbList — schema.org Type",
      "url": "https://schema.org/BreadcrumbList",
      "type": "spec",
      "publisher": "schema.org",
      "verified": "2026-08-20",
      "keyFindings": "Core (non-pending) type: 'An ItemList consisting of a chain of linked Web pages, typically described using at least their URL and their name, and typically ending with the current page.' Order reconstructed from the 'position' property."
    },
    {
      "id": "schema-buyaction",
      "title": "BuyAction — schema.org Type",
      "url": "https://schema.org/BuyAction",
      "type": "spec",
      "publisher": "schema.org",
      "verified": "2026-08-20",
      "keyFindings": "Core type: 'The act of giving money to a seller in exchange for goods or services rendered.' Hierarchy Thing > Action > TradeAction > BuyAction. Properties: seller, price, priceCurrency, priceSpecification, plus inherited Action target/EntryPoint. Reported usage 1M–10M domains (Google July 2026 index aggregation) — but no AI or shopping-agent vendor documents consuming it."
    },
    {
      "id": "schema-contactpoint",
      "title": "ContactPoint — schema.org Type",
      "url": "https://schema.org/ContactPoint",
      "type": "spec",
      "publisher": "schema.org",
      "verified": "2026-08-20",
      "keyFindings": "Core type, usage reported at 1M–10M domains. Properties: telephone, email, contactType, areaServed, contactOption, hoursAvailable, faxNumber, productSupported. Used by Organization, Person, JobPosting (applicationContact), ServiceChannel."
    },
    {
      "id": "schema-datafeed",
      "title": "schema.org DataFeed",
      "url": "https://schema.org/DataFeed",
      "type": "spec",
      "publisher": "schema.org",
      "verified": "2026-08-20",
      "keyFindings": "DataFeed = 'a single feed providing structured information about one or more entities or topics'; hierarchy Thing > CreativeWork > Dataset > DataFeed; primary property dataFeedElement accepting DataFeedItem/Text/Thing; DataFeedItem examples use dateCreated, dateModified, item. Adoption is only 1K-10K domains per Google's web index (July 2026 aggregation) — too thin to score against."
    },
    {
      "id": "schema-faqpage",
      "title": "schema.org — FAQPage",
      "url": "https://schema.org/FAQPage",
      "type": "spec",
      "publisher": "schema.org",
      "verified": "2026-08-21",
      "keyFindings": "\"A FAQPage is a WebPage presenting one or more 'Frequently asked questions'\". Adoption reported at 1M–10M domains."
    },
    {
      "id": "schema-person",
      "title": "schema.org — Person",
      "url": "https://schema.org/Person",
      "type": "spec",
      "publisher": "schema.org",
      "verified": "2026-08-21",
      "keyFindings": "Defines `name`, `jobTitle`, `affiliation` and `sameAs` but marks none required; `affiliation` is \"An organization that this person is affiliated with\"."
    },
    {
      "id": "schema-searchaction",
      "title": "schema.org — SearchAction",
      "url": "https://schema.org/SearchAction",
      "type": "spec",
      "publisher": "schema.org",
      "verified": "2026-08-21",
      "keyFindings": "\"The act of searching for an object\" — used as a `potentialAction` on `WebSite` with an `EntryPoint` `urlTemplate` carrying the query placeholder."
    },
    {
      "id": "schemaorg-about",
      "title": "About schema.org",
      "url": "https://schema.org/docs/about.html",
      "type": "spec",
      "publisher": "schema.org",
      "verified": "2026-08-20",
      "keyFindings": "Founded and sponsored by Google, Microsoft, Yahoo and Yandex; a \"collaborative, community activity\". Confirms the vocabulary's governance and its four founding search-engine consumers. It is a community vocabulary, not a ratified standards-body specification."
    },
    {
      "id": "schemaorg-articlebody",
      "title": "schema.org/articleBody",
      "url": "https://schema.org/articleBody",
      "type": "spec",
      "publisher": "Schema.org",
      "verified": "2026-08-20",
      "keyFindings": "articleBody: \"The actual body of the article\", expected type Text, used on Article; reported in use across 1M-10M domains. Confirms the specific property that, when populated, duplicates the entire visible article inside a <script type=\"application/ld+json\"> block."
    },
    {
      "id": "schemaorg-confirmaction",
      "title": "schema.org: ConfirmAction",
      "url": "https://schema.org/ConfirmAction",
      "publisher": "schema.org",
      "type": "spec",
      "keyFindings": "Type exists and is active, but usage is '< 1K Domains' based on Google's web index, July 2026 — the lowest adoption figure of any signal measured across these four audits. ReserveAction (https://schema.org/ReserveAction) shows 10K-100K domains.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "schemaorg-howto",
      "title": "schema.org: HowTo",
      "url": "https://schema.org/HowTo",
      "publisher": "schema.org",
      "type": "spec",
      "keyFindings": "Type is active — not deprecated, not attic, not pending. Usage: 100K - 1M domains (Google web index, July 2026). Present 12 times in the official schemaorg-current-https.jsonld vocabulary dump.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "schemaorg-potentialaction",
      "title": "schema.org: potentialAction",
      "url": "https://schema.org/potentialAction",
      "publisher": "schema.org",
      "type": "spec",
      "keyFindings": "Active, not deprecated. 10M+ domains (Google web index, July 2026) — but dominated by CMS-emitted WebSite/SearchAction boilerplate for the now-removed sitelinks search box. OrderAction: 10K-100K domains. ReserveAction: 10K-100K domains.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "schemaorg-sameas",
      "title": "schema.org: sameAs property",
      "url": "https://schema.org/sameAs",
      "type": "spec",
      "publisher": "schema.org (W3C Schema.org Community Group)",
      "verified": "2026-08-20",
      "keyFindings": "Official definition, verbatim: 'URL of a reference Web page that unambiguously indicates the item's identity. E.g. the URL of the item's Wikipedia page, Wikidata entry, or official website.' Expected type: URL. Domain: Thing (so applies to both Person and Organization). Ratified vocabulary term; establishes the identity-disambiguation mechanism, though the spec itself names no AI-engine consumer."
    },
    {
      "id": "schemaorg-shipping",
      "title": "schema.org OfferShippingDetails",
      "url": "https://schema.org/OfferShippingDetails",
      "type": "spec",
      "publisher": "schema.org",
      "verified": "2026-08-20",
      "keyFindings": "Properties: deliveryTime (ShippingDeliveryTime — carries handlingTime and transitTime), shippingRate (MonetaryAmount or ShippingRateSettings), shippingDestination (DefinedRegion), shippingOrigin (DefinedRegion), doesNotShip (Boolean), hasShippingService (ShippingService), weight, height, width, depth, validForMemberTier. Note handlingTime/transitTime are nested under deliveryTime, not direct properties — an auditor must traverse offers.shippingDetails.deliveryTime.{handlingTime,transitTime}."
    },
    {
      "id": "schemaorg-sitenavigationelement",
      "title": "SiteNavigationElement",
      "url": "https://schema.org/SiteNavigationElement",
      "publisher": "schema.org",
      "type": "spec",
      "keyFindings": "Active schema.org type for \"a navigation element of the page\"; inherits from WebPageElement/CreativeWork; key properties name, url, position. Adoption reported as \"1M - 10M Domains Based on monthly aggregations from Google's web index\" (July 2026). This is the standardized, widely deployed machine-readable navigation signal that navigation.json duplicates without any consumer.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "schemaorg-speakable",
      "title": "schema.org/speakable",
      "url": "https://schema.org/speakable",
      "type": "spec",
      "publisher": "schema.org",
      "verified": "2026-08-21",
      "keyFindings": "Property defined for marking sections \"especially appropriate for text-to-speech conversion\"; supports id-references, CSS selectors and XPath via SpeakableSpecification. Usage reported at 100K–1M domains (source: Google's web index, July 2026). Google is the only named consumer."
    },
    {
      "id": "schemaorg-vocab-dump",
      "title": "schema.org current vocabulary (schemaorg-current-https.jsonld)",
      "url": "https://schema.org/version/latest/schemaorg-current-https.jsonld",
      "publisher": "schema.org",
      "type": "spec",
      "keyFindings": "Downloaded (1.5 MB) and grepped. schema:ContactAction = 0 occurrences. schema:BookAction = 0 occurrences. schema:OrderAction = 2. schema:ReserveAction = 1. schema:ConfirmAction = 1. Confirms ContactAction and BookAction are not schema.org types; https://schema.org/ContactAction and https://schema.org/BookAction both return HTTP 404.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "screamingfrog-seo-spider",
      "title": "Screaming Frog SEO Spider",
      "url": "https://www.screamingfrog.co.uk/seo-spider/",
      "type": "vendor-doc",
      "publisher": "Screaming Frog",
      "verified": "2026-08-20",
      "keyFindings": "The incumbent technical-SEO crawler. Its AI features are AI-assisted analysis of the crawl (custom prompts via OpenAI, Gemini, Ollama, Anthropic; vector-embedding near-duplicate detection), not AI-readiness auditing: the product page describes no AI-crawler access checks, no llms.txt analysis, and no agent-readiness surface. Confirms the incumbent audit tools have not annexed this category yet."
    },
    {
      "id": "searchviu-schema-ai-fetch-test",
      "title": "Schema Markup and AI in 2025: What ChatGPT, Claude, Perplexity & Gemini Really See",
      "url": "https://www.searchviu.com/en/schema-markup-and-ai-in-2025-what-chatgpt-claude-perplexity-gemini-really-see/",
      "type": "study",
      "publisher": "searchVIU",
      "verified": "2026-08-21",
      "keyFindings": "Controlled test page (\"GEO Bears\", 8 variants) placing prices exclusively in different layers: visible HTML, JS-rendered DOM, JSON-LD, hidden/visible Microdata, hidden/visible RDFa. Tested ChatGPT, Claude, Gemini, Perplexity, Google AI Mode. Finding: \"JSON-LD Schema is NOT read by AI chatbots during direct fetch\" — the price present only in JSON-LD was not retrieved by any of the 5 systems. Success rates: Gemini 4/8 (only system rendering JS on live fetch), ChatGPT 3/8, Google AI Mode 2/8, Perplexity 1/8, Claude 0/8. Author caveat: schema may still be used in indexing/training phases or by index-backed systems (AI Overviews, Copilot)."
    },
    {
      "id": "security-txt-prevalence-study",
      "title": "security.txt Revisited: Analysis of Prevalence and Conformity in 2022",
      "url": "https://seclab.cs.hm.edu/assets/pdf/th-sectxt-2023.pdf",
      "type": "study",
      "publisher": "Hochschule München / ACM Digital Threats: Research and Practice",
      "verified": "2026-08-20",
      "keyFindings": "Longitudinal measurement of security.txt adoption and RFC conformity across the top one million websites. Corroborating later measurements put adoption at ~0.7% of the top 1M in April 2024 rising to ~1.25% in 2025, with low RFC-9116 conformity rates. NOT fetched directly during this research — cite with the ACM DOI 10.1145/3609234 as the canonical reference."
    },
    {
      "id": "seer-recency-study",
      "title": "Study: AI Brand Visibility and Content Recency",
      "url": "https://www.seerinteractive.com/insights/study-ai-brand-visibility-and-content-recency",
      "type": "study",
      "publisher": "Seer Interactive",
      "verified": "2026-08-21",
      "keyFindings": "5,000+ URLs with extractable publish dates, cited across ChatGPT, Perplexity and AI Overviews; primary metric was AI log-file hits from three ChatGPT bots, supplemented with Peec.ai citation data; content spanned 2004–2025; published 25 June 2025. Log hits: ~65% to content from the past year, 79% from the last two years, 89% from the last three, only 6% older than six years. Citation share from 2023–2025: ChatGPT 71%, Perplexity ~80%, AI Overviews ~85%. Important counter-finding: strong industry variation — Financial Services showed extreme recency bias while Energy and decking (instructional) content showed 10–15-year-old pages still drawing AI bot activity, indicating evergreen content retains value; the study concludes query intent matters more than mechanical recency optimization. NOTE: its AI Overviews recency figure conflicts with the Ahrefs 17M-citation dataset."
    },
    {
      "id": "sej-bing-geo-guidelines",
      "title": "Bing Adds GEO To Official Guidelines, Expands AI Abuse Definitions",
      "url": "https://www.searchenginejournal.com/bing-adds-geo-to-official-guidelines-expands-ai-abuse-definitions/568442/",
      "type": "article",
      "publisher": "Search Engine Journal",
      "verified": "2026-08-20",
      "keyFindings": "Reports Microsoft's rewrite of the Bing Webmaster Guidelines to cover Copilot grounding and citations, introducing GEO as a recognized concept and stating that GEO does not guarantee citations any more than SEO guarantees rankings. Quoted guidance relevant to this domain: 'Facts should be stated directly rather than implied, since AI systems need content that can be verified independently'; 'Entity names should be clear and consistent, with no ambiguous references'; 'Single-topic pages are more likely to be selected for grounding results'; 'Focus each URL on a single topic and place essential information near the top.' This is the only vendor statement located that endorses answer-first placement."
    },
    {
      "id": "sej-google-llms-txt-speculative",
      "title": "Google Confirms LLMs.txt Has No Current Implementation / 'purely speculative for now'",
      "url": "https://www.searchenginejournal.com/google-says-llms-txt-is-purely-speculative-for-now/577576/",
      "type": "article",
      "publisher": "Search Engine Journal",
      "verified": "2026-08-20",
      "keyFindings": "John Mueller (Google), reported 2026-06-02: 'I don't think anyone knows – it's purely speculative for now (the file has existed for years, yet none of the AI systems use it — what does it mean?).' Also: 'When an AI platform that brings you clients complains that it needs the file for your site, then I'd recommend taking the time to create one.' Mueller prefers WebMCP: 'they have clear goals & processes.' And: 'the most basic agentic optimization... don't block agents.'"
    },
    {
      "id": "sej-lighthouse-markdown-links",
      "title": "Lighthouse Fails Your llms.txt Without Markdown Links",
      "url": "https://www.searchenginejournal.com/lighthouse-fails-your-llms-txt-without-markdown-links/577590/",
      "type": "article",
      "publisher": "Search Engine Journal",
      "verified": "2026-08-20",
      "keyFindings": "Published 2026-07-03. Documents a real failure: a plain-text llms.txt with working bare URLs failed with 'File does not appear to contain any links.' until links were wrapped in [text](url). Key honest caveat: 'The audit checks whether your file is mechanically parseable. It does not check whether the file describes your website usefully.'"
    },
    {
      "id": "sel-google-llms-txt-no-effect",
      "title": "Google says llms.txt files won't harm or help your search rankings",
      "url": "https://searchengineland.com/google-says-llms-txt-files-wont-harm-or-help-your-search-rankings-480264",
      "type": "article",
      "publisher": "Search Engine Land",
      "verified": "2026-08-20",
      "keyFindings": "Reports the 2026-06-15 Google docs update that added the LLMS.txt mythbusting entry, pinpointing the URL developers.google.com/search/docs/fundamentals/ai-optimization-guide#mythbusting. Also relays Gary Illyes at Google Search Central Live: Google does not support llms.txt and has no plans to."
    },
    {
      "id": "semantic-chunking-cost-arxiv",
      "title": "Is Semantic Chunking Worth the Computational Cost?",
      "url": "https://arxiv.org/abs/2410.13070",
      "type": "study",
      "publisher": "Qu, Tu, Bao (arXiv, October 2024)",
      "verified": "2026-08-20",
      "keyFindings": "Counter-evidence for the 'chunkability' signal: across document retrieval, evidence retrieval, and retrieval-based answer generation, 'the computational costs associated with semantic chunking are not justified by consistent performance gains.' Semantically coherent segmentation did not reliably beat naive fixed-size splitting — implying retrieval pipelines often do not honor a publisher's semantic boundaries at all, weakening the causal path from authored chunk structure to retrieval outcome."
    },
    {
      "id": "semrush",
      "title": "Semrush AI Visibility Toolkit",
      "url": "https://www.semrush.com/features/ai-visibility/",
      "type": "vendor-doc",
      "publisher": "Semrush",
      "verified": "2026-08-20",
      "keyFindings": "Visibility Overview, Brand Performance, Competitor Research, Prompt Tracking, AI-Cited Media, Prompt Research. The page cross-sells the separate classic Site Audit for 'technical health' but documents no AI-bot-specific crawlability, llms.txt or agent-schema checks inside the AI toolkit."
    },
    {
      "id": "semrush-ai-overviews-study",
      "title": "We Studied 200,000 AI Overviews: Here's What We Learned",
      "url": "https://www.semrush.com/blog/ai-overviews-study/",
      "type": "dataset",
      "publisher": "Semrush",
      "verified": "2026-08-21",
      "keyFindings": "200,000 US keywords (100k desktop, 100k mobile), AI Overviews captured September 1–10, 2024. AI Overview length averages 119 words desktop / 91 mobile (range 5–417). Average 11 links per AI Overview. Only 20–26% of AIO links matched the top-10 organic results. Informational intent dominates (80% desktop, 76% mobile); 'how' and 'what' questions are the most common question types (35% desktop, 32% mobile). Important limitation for this domain: the study reports the FORMAT OF THE OUTPUT (AI Overviews frequently render as lists) and provides no data on on-page input structure requirements — a distinction routinely conflated in secondary SEO coverage."
    },
    {
      "id": "semrush-most-cited-domains",
      "title": "The Most-Cited Domains in AI: A 3-Month Study",
      "url": "https://www.semrush.com/blog/most-cited-domains-ai/",
      "type": "study",
      "publisher": "Semrush",
      "verified": "2026-08-20",
      "keyFindings": "230,000+ prompts across ChatGPT Search, Google AI Mode and Perplexity; 100M+ AI citations tracked; 14 July – 12 October 2025. Top cited domains overall: Reddit, Wikipedia, LinkedIn, YouTube, Forbes. Extreme volatility on ChatGPT: Reddit collapsed from ~60% of responses in early August to ~10% by mid-September; Wikipedia fell from ~55% to under 20%. Google AI Mode was stable and prefers Google-owned/partnered properties (LinkedIn ~15%, Wikipedia ~2%). The study attributes shifts to deliberate platform-level source rebalancing and manipulation-resistance, NOT to authority metrics, freshness or page-level content quality — strong counter-evidence that on-page authority signals drive citation selection."
    },
    {
      "id": "semrush-technical-seo-ai",
      "title": "How Do Technical SEO Factors Impact AI Search? [Study]",
      "url": "https://www.semrush.com/blog/technical-seo-impact-on-ai-search-study/",
      "type": "study",
      "publisher": "Semrush",
      "verified": "2026-08-20",
      "keyFindings": "5 million cited URLs across ChatGPT Search and Google AI Mode. Schema presence among cited pages: Organization 25% (ChatGPT) / 34% (AI Mode); Article 20% / 26%; Breadcrumb 15% / 20%; Open Graph and JSON-LD on the majority of cited pages. URL structure: citation peak at 21–25 character slugs (~87K citations), secondary peak at 6–10 characters (~57K), with moderate slug lengths (17–40 chars) consistently outperforming very short or very long URLs. Cited pages also showed higher visits, longer sessions, more pages/visit. Explicitly correlational, not causal. Notably NOT tested: internal linking structure, author markup, publication/modification dates, page speed, heading hierarchy — and the slug data is a distribution over already-cited URLs with no uncited baseline."
    },
    {
      "id": "seoclarity",
      "title": "seoClarity Clarity ArcAI suite",
      "url": "https://www.seoclarity.net/",
      "type": "vendor-doc",
      "publisher": "seoClarity",
      "verified": "2026-08-20",
      "keyFindings": "12 named modules: Track Visibility, Research Prompts, Analyze Sentiment, Optimize Content, Measure Performance, Discover Bot Activity (/ai-seo/ai-bot-activity-tracking — 'know if AI bots access your pages', log-based), Monitor Accuracy, MCP Server and API, Accelerate Indexation, Monitor Web Mentions, Track AI Shopping, Product Feed Optimizer. Bot Activity is observational, not a conformance audit."
    },
    {
      "id": "server-rendered-audit",
      "title": "agent-lighthouse server-rendered.ts (audit 8.13)",
      "url": "https://github.com/ForkPoint/agent-lighthouse/blob/main/packages/core/src/audits/technical-readiness/server-rendered.ts",
      "publisher": "ForkPoint / agent-lighthouse",
      "type": "repo",
      "verified": "2026-08-20",
      "keyFindings": "Sibling audit at critical priority that directly measures the delivered HTML ('Homepage <main> has > 50 words or > 200 characters of text content') and fails client-side-only pages. This is the ground-truth measurement that framework detection merely proxies, making 8.21 redundant. Read from the local working copy at /Users/kirov/dev/forkpoint/agent-lighthouse/packages/core/src/audits/technical-readiness/server-rendered.ts; the GitHub URL was not fetched.",
      "accessed": "2026-08-21"
    },
    {
      "id": "shopware-ai-catalog-twig",
      "title": "Shopware core ai-catalog.json.twig",
      "url": "https://raw.githubusercontent.com/shopware/shopware/trunk/src/Core/Framework/Resources/views/files/agentic/.well-known/ai-catalog.json.twig",
      "publisher": "Shopware",
      "type": "repo",
      "keyFindings": "Shopware core ships a first-party template emitting specVersion/host/entries with an MCP server-card entry per sales channel — platform-level adoption, not a one-off site.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "shopware-twig-fields",
      "title": "Shopware core ai-catalog.json.twig field set",
      "url": "https://raw.githubusercontent.com/shopware/shopware/trunk/src/Core/Framework/Resources/views/files/agentic/.well-known/ai-catalog.json.twig",
      "publisher": "Shopware",
      "type": "repo",
      "keyFindings": "Platform template emits identifier, displayName, type, url, description, tags, capabilities, representativeQueries — matching the spec, not the audit's field list.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "sitemaps-org",
      "title": "Sitemaps XML format — protocol",
      "url": "https://www.sitemaps.org/protocol.html",
      "type": "spec",
      "publisher": "sitemaps.org",
      "verified": "2026-08-20",
      "keyFindings": "lastmod must be W3C Datetime (YYYY-MM-DD or full timestamp). Path-scope rule: a sitemap at /catalog/sitemap.xml may only list URLs under /catalog/; all URLs must share protocol and host with the sitemap. 50,000 URLs / 50MB (52,428,800 bytes) per file; index files limited to 50,000 sitemaps and may only reference sitemaps on the same site."
    },
    {
      "id": "sitemaps-org-protocol",
      "title": "Sitemaps XML format — sitemaps.org protocol 0.9",
      "url": "https://www.sitemaps.org/protocol.html",
      "type": "spec",
      "publisher": "sitemaps.org",
      "verified": "2026-08-20",
      "keyFindings": "Documents 'Sitemap: http://www.example.com/sitemap.xml' in robots.txt, multiple entries permitted, and 'This directive is independent of the user-agent line, so it doesn't matter where you place it in your file.' Protocol version 0.9. The Sitemap robots.txt line is an out-of-band convention, not part of RFC 9309 — RFC 9309 §2.2.4 only permits crawlers to interpret it."
    },
    {
      "id": "sitemaps-protocol",
      "title": "Sitemaps XML format (protocol 0.9)",
      "url": "https://www.sitemaps.org/protocol.html",
      "type": "spec",
      "publisher": "sitemaps.org",
      "verified": "2026-08-20",
      "keyFindings": "Namespace http://www.sitemaps.org/schemas/sitemap/0.9; UTF-8; max 50,000 URLs and 50MB per file with index files for larger sets. Optional <lastmod> 'must be set to the date the linked page was last modified, not when the sitemap is generated.' Discovery via the robots.txt `Sitemap:` directive."
    },
    {
      "id": "slack-link-unfurling",
      "title": "Slack — unfurling links in messages",
      "url": "https://docs.slack.dev/messaging/unfurling-links-in-messages",
      "type": "vendor-doc",
      "publisher": "Slack",
      "verified": "2026-08-21",
      "keyFindings": "\"Slack crawls the URL, looks for common OpenGraph and X (formerly known as Twitter) Card metadata, and renders some micro-approximation of the content.\""
    },
    {
      "id": "smithery-registry",
      "title": "Smithery — MCP server registry",
      "url": "https://smithery.ai/",
      "type": "dataset",
      "publisher": "Smithery",
      "verified": "2026-08-20",
      "keyFindings": "Registry advertises 16,671+ MCP servers browsable. Independent count roughly 25% below PulseMCP's 22,033, illustrating that MCP 'server count' varies materially by directory inclusion criteria."
    },
    {
      "id": "sparktoro-similarweb-ai-brand-mentions",
      "title": "New Research from Similarweb: How AI Brand Mentions Influence Direct Visits & Traditional Search Queries",
      "url": "https://sparktoro.com/blog/new-research-from-similarweb-how-ai-brand-mentions-influence-direct-visits-traditional-search-queries/",
      "type": "study",
      "publisher": "SparkToro / Similarweb",
      "verified": "2026-08-20",
      "keyFindings": "Published 2026-06-29. Similarweb clickstream panel across finance, travel and beauty; measures behavior in the ~7 days after an AI recommendation. When AI recommends a brand, visitors are 7.2% more likely to visit American Express and 14.2% more likely for Capital One. Post-AI behavior shifts from search to direct: search ~15% lower, direct higher. Explicit caveat from Fishkin: findings cover big brands (Sephora, Amex, Capital One) and may not generalize to smaller brands; no conversion-rate measurement."
    },
    {
      "id": "specification-website-link",
      "title": "specification.website live HTML/Link header",
      "url": "https://specification.website/",
      "publisher": "Joost de Valk (specification.website)",
      "type": "article",
      "keyFindings": "Serves <link rel=\"ai-catalog\" type=\"application/ai-catalog+json\" title=\"Agentic Resource Discovery catalog\" href=\"/.well-known/ai-catalog.json\"> plus the matching HTTP Link header — proof the rel token, not rel=alternate, is what conformant sites emit.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "specification-website-llms-txt",
      "title": "The Website Specification — /llms.txt page",
      "url": "https://specification.website/spec/agent-readiness/llms-txt/",
      "type": "draft-spec",
      "publisher": "specification.website",
      "verified": "2026-08-20",
      "keyFindings": "Model of honest counter-evidence in a competitor: marks /llms.txt status 'recommended', states 'It is not a ratified standard', and warns 'No major model vendor has committed to consuming it', framing it as 'a low-cost bet'. Its Common mistakes section names 'Assuming any specific model uses it today. Adoption is uneven and not always disclosed.' Cites llmstxt.org and isitagentready.com as sources. Page shape: definition -> rationale -> implementation -> pitfalls -> verification. Proves published counter-evidence is achievable, and sets the bar we must clear."
    },
    {
      "id": "specification-website-repo",
      "title": "The Website Specification (repo)",
      "url": "https://github.com/jdevalk/specification.website",
      "type": "draft-spec",
      "publisher": "Joost de Valk (Yoast founder) — 834 stars, MIT code / CC BY 4.0 content, actively pushed 2026-08-20",
      "verified": "2026-08-20",
      "keyFindings": "'Platform-agnostic, sourced on every page, written for humans and agents.' Ten categories including a dedicated Agent Readiness section; also ships an MCP server at mcp.specification.website/mcp. Self-describes as 'Not opinion. Where there is no settled standard, the spec says so.' The most credible evidence-first competitor by adoption (834 stars vs 0-3 for the audit-tool long tail) — but it is a normative spec, not a scanner: no score, no weights, no automated audit run."
    },
    {
      "id": "statcounter-ai-chatbot-share",
      "title": "AI Chatbot Market Share Worldwide — Statcounter GlobalStats",
      "url": "https://gs.statcounter.com/ai-chatbot-market-share",
      "type": "dataset",
      "publisher": "Statcounter",
      "verified": "2026-08-20",
      "keyFindings": "July 2026 snapshot: ChatGPT 77.92%, Google Gemini 9.9%, Perplexity 5.88%, Claude 3.19%, Microsoft Copilot 3.07%, DeepSeek 0.03%. Filterable by device, region and time. Methodology is tracking-code-based referral/visit measurement, which systematically under-measures assistants that suppress referrers and in-app usage — treat as directional ranking, not a usage census."
    },
    {
      "id": "statcounter-browser-share",
      "title": "Browser Market Share Worldwide — Statcounter GlobalStats",
      "url": "https://gs.statcounter.com/browser-market-share",
      "type": "dataset",
      "publisher": "Statcounter",
      "verified": "2026-08-20",
      "keyFindings": "July 2026: Chrome 68.22%, Safari 16.47%, Edge 5.37%, Firefox 3.34%, Samsung Internet 2.06%, Opera 1.88%. NEGATIVE RESULT: no agentic/AI browser (Perplexity Comet, OpenAI Atlas) appears in the dataset at all — none has reached measurable share in the largest public browser panel."
    },
    {
      "id": "stimulus-data-action",
      "title": "Stimulus Reference — Actions",
      "url": "https://stimulus.hotwired.dev/reference/actions",
      "publisher": "Hotwired / Basecamp",
      "type": "vendor-doc",
      "keyFindings": "data-action in Stimulus is an event-to-method binding: syntax `event->controller#method`, e.g. \"click->gallery#next\", with keyboard filters (\"keydown.esc->modal#close\"), global targets (\"resize@window->gallery#layout\") and options (\":prevent\", \":!passive\"). Ships with Rails. Any Hotwire site is saturated with data-action attributes carrying zero agent semantics, making the audit's presence check unsound.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "stripe-acp-docs",
      "title": "Agentic Commerce Protocol | Stripe Documentation",
      "url": "https://docs.stripe.com/agentic-commerce/acp",
      "type": "vendor-doc",
      "publisher": "Stripe",
      "verified": "2026-08-20",
      "keyFindings": "Stripe's implementation reference for ACP, the open standard defining how AI agents interact with businesses to complete purchases. Confirms the feed + merchant-endpoint architecture rather than crawlable cart URLs."
    },
    {
      "id": "tdmrep-cg-final",
      "title": "TDM Reservation Protocol — Community Group Final Report",
      "url": "https://www.w3.org/community/reports/tdmrep/CG-FINAL-tdmrep-20240510",
      "type": "draft-spec",
      "publisher": "W3C Community Group",
      "verified": "2026-08-21",
      "keyFindings": "Final Report of 10 May 2024. Its own status section states: \"It is not a W3C Standard nor is it on the W3C Standards Track.\" Defines the three signalling methods the audit looks for."
    },
    {
      "id": "tdmrep-community-group",
      "title": "W3C TDM Reservation Protocol Community Group",
      "url": "https://www.w3.org/community/tdmrep",
      "type": "draft-spec",
      "publisher": "W3C Community Group",
      "verified": "2026-08-21",
      "keyFindings": "Adoption is publisher-side, not crawler-side: names Mondadori, Penguin Random House, the STM association, Copyright Clearance Center, Taylor & Francis and the BBC among participants."
    },
    {
      "id": "tiktoken",
      "title": "openai/tiktoken",
      "url": "https://github.com/openai/tiktoken",
      "type": "repo",
      "publisher": "OpenAI",
      "verified": "2026-08-20",
      "keyFindings": "Fast BPE tokenizer with cl100k_base and o200k_base encodings and encoding_for_model(); counts tokens fully offline, 3-6x faster than comparable tokenizers. Makes every token metric in this domain deterministic, reproducible and CI-friendly with no network or model call."
    },
    {
      "id": "tollbit-robots-noncompliance",
      "title": "15% of AI page fetchers in Europe reached disallowed URLs, TollBit finds",
      "url": "https://ppc.land/15-of-ai-page-fetchers-in-europe-reached-disallowed-urls-tollbit-finds/",
      "type": "study",
      "publisher": "PPC Land, reporting TollBit 'State of the Bots' H1 2026 ('The Pipes are Leaky')",
      "verified": "2026-08-20",
      "keyFindings": "Published 2026-08-14. 'Roughly 15% of identified page-fetching agents' reached disallowed URLs on European sites. 'ChatGPT-User, Bytespider and Youbot each accessed disallowed pages on nearly half of the European sites that had explicitly listed them.' ChatGPT-User 'reached disallowed pages on more sites than any other bot' while also being 'disallowed by more sites than any other bot of its type'. Regional blocking rates: Claude-User 9% EU vs 26% NA; Perplexity-User 13% EU vs 26% NA. This is the primary counter-evidence that a robots.txt disallow is not self-enforcing."
    },
    {
      "id": "tollbit-state-of-the-bots",
      "title": "TollBit State of the Bots reports",
      "url": "https://tollbit.com/bots/25q1",
      "type": "dataset",
      "publisher": "TollBit",
      "verified": "2026-08-20",
      "keyFindings": "URL resolves; report series index confirms editions: 2026 Q1 & Q2 'The Bad Bots', 2025 Q3 & Q4 'The Leaky Pipes', 2025 Q2 'Good Net Citizens', 2025 Q1 'The Rise of RAG Bots', 2024 Q4 'The First Issue'. Sections cover the scraper ecosystem & proxy networks and referral traffic. Headline scrape volumes, robots.txt compliance rates and scrape-to-referral ratios are rendered behind the report pages and were not machine-extractable in this pass — cite only after reading a specific dated edition."
    },
    {
      "id": "trafilatura-corefunctions",
      "title": "trafilatura core functions documentation",
      "url": "https://trafilatura.readthedocs.io/en/latest/corefunctions.html",
      "type": "vendor-doc",
      "publisher": "Adrien Barbaresi / trafilatura",
      "verified": "2026-08-21",
      "keyFindings": "baseline() 'tries a series of sources and takes the first that yields enough text: JSON content embedded in scripts or attributes (schema.org properties…), article tags, text paragraphs, schema.org teaser descriptions, and finally the raw text of the whole page body' — an explicit preference ladder that rewards <article>. include_tables is on by default; include_formatting 'keep[s] structural elements related to formatting (kept in XML, rendered as markdown for text formats)'. try_readability() and try_justext() are documented fallbacks."
    },
    {
      "id": "trafilatura-eval",
      "title": "Trafilatura — evaluation of web content extractors",
      "url": "https://trafilatura.readthedocs.io/en/latest/evaluation.html",
      "type": "study",
      "publisher": "Adrien Barbaresi / trafilatura docs",
      "verified": "2026-08-20",
      "keyFindings": "Benchmark over 990 documents (run dated 2026-08-04): trafilatura 2.2.0 F=0.924 (P 0.906 / R 0.943), magic-html F=0.889, news-please F=0.836, readability-lxml F=0.826, goose3 F=0.810 with precision 0.936 but recall 0.714, inscriptis recall 0.991 with precision 0.534. Extractors disagree massively on what the main content of a page is — quantified spread that justifies an extractor-agreement metric."
    },
    {
      "id": "trafilatura-settings",
      "title": "trafilatura/settings.py — MANUALLY_CLEANED / MANUALLY_STRIPPED",
      "url": "https://raw.githubusercontent.com/adbar/trafilatura/master/trafilatura/settings.py",
      "publisher": "Adrien Barbaresi / trafilatura",
      "type": "repo",
      "keyFindings": "MANUALLY_CLEANED begins with \"aside\", followed by footer, form, iframe, menu, nav, script etc. — <aside> subtrees are removed as boilerplate before extraction. trafilatura is a standard HTML-to-text extractor in LLM corpus construction, so aside content is systematically excluded from text that trains and grounds models.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "trafilatura-settings-address",
      "title": "trafilatura/settings.py — MANUALLY_STRIPPED",
      "url": "https://raw.githubusercontent.com/adbar/trafilatura/master/trafilatura/settings.py",
      "publisher": "Adrien Barbaresi / trafilatura",
      "type": "repo",
      "keyFindings": "MANUALLY_STRIPPED contains \"address\" ( \"abbr\", \"acronym\", \"address\", \"bdi\", \"bdo\", ... ) — the <address> tag is unwrapped and discarded during extraction, so downstream LLM text retains the contact text with no marker that it was <address>. Positive evidence the signal is destroyed before reaching a model.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "trafilatura-xpaths",
      "title": "trafilatura/xpaths.py (BODY_XPATH, OVERALL_DISCARD_XPATH)",
      "url": "https://raw.githubusercontent.com/adbar/trafilatura/master/trafilatura/xpaths.py",
      "type": "vendor-doc",
      "publisher": "Adrien Barbaresi / trafilatura",
      "verified": "2026-08-20",
      "keyFindings": "Hard proof that landmark elements drive main-content extraction in the library most used to build LLM web corpora. BODY_XPATH selects on 'self::article or self::div or self::main or self::section', plus @itemprop='articleBody', @role='article', @id='article'/'story'. OVERALL_DISCARD_XPATH removes nodes whose role contains 'nav' (case-normalised), footer/header markers, comment sections, and @aria-hidden='true'."
    },
    {
      "id": "trojan-source",
      "title": "Trojan Source: Invisible Vulnerabilities",
      "url": "https://trojansource.codes/",
      "type": "study",
      "publisher": "University of Cambridge (Boucher & Anderson)",
      "verified": "2026-08-20",
      "keyFindings": "Unicode bidirectional control characters reorder tokens at the encoding level so the rendered order differs from the logical order a parser reads; homoglyph variant defines confusable identifiers. CVE-2021-42574 (bidi) and CVE-2021-42694 (homoglyph). Directly transferable to agent text extraction, which reads logical order while the human reads rendered order."
    },
    {
      "id": "turndown-commonmark-rules",
      "title": "turndown — commonmark-rules.js",
      "url": "https://raw.githubusercontent.com/mixmark-io/turndown/master/src/commonmark-rules.js",
      "type": "repo",
      "publisher": "mixmark-io",
      "verified": "2026-08-21",
      "keyFindings": "The `fencedCodeBlock` rule derives the fence language from the code element class: `const className = node.firstChild.getAttribute('class') || ''`."
    },
    {
      "id": "twitter-cards-markup-archived",
      "title": "Cards Markup Tag Reference (archived, Dec 2023)",
      "url": "https://web.archive.org/web/20231229075931/https://developer.twitter.com/en/docs/twitter-for-websites/cards/overview/markup",
      "type": "vendor-doc",
      "publisher": "Twitter/X Developer Platform via Internet Archive",
      "verified": "2026-08-20",
      "keyFindings": "Last accessible version of the Cards reference. It documents an OpenGraph fallback column for every content-bearing tag: twitter:title -> og:title, twitter:description -> og:description, twitter:image -> og:image, twitter:image:alt -> og:image:alt, twitter:card -> og:type, and \"If an og:type, og:title and og:description exist in the markup but twitter:card is absent, then a summary card may be rendered.\" Only X-specific fields (twitter:site, twitter:creator, player, app) have no OG equivalent. Establishes that twitter:* content tags are strictly redundant with Open Graph."
    },
    {
      "id": "unicode-tags",
      "title": "Hiding and Finding Text with Unicode Tags",
      "url": "https://embracethered.com/blog/posts/2024/hiding-and-finding-text-with-unicode-tags/",
      "type": "article",
      "publisher": "Embrace The Red (Johann Rehberger)",
      "verified": "2026-08-20",
      "keyFindings": "The Unicode Tags block (U+E0000–U+E007F) mirrors ASCII and, per UTS #51, renders as nothing in tag-unaware implementations. Modern LLM tokenizers handle these codepoints, so an invisible ASCII payload survives copy/paste and human review and is read by the model. Demonstrated ChatGPT acting on tag-encoded instructions. Recommends filtering the range at both prompt and response time."
    },
    {
      "id": "vercel-ai-crawler",
      "title": "The rise of the AI crawler",
      "url": "https://vercel.com/blog/the-rise-of-the-ai-crawler",
      "type": "study",
      "publisher": "Vercel",
      "verified": "2026-08-21",
      "keyFindings": "\"none of the major AI crawlers currently render JavaScript\" — explicitly GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot — though they do fetch JS files as text (ChatGPT 11.50%, Claude 23.84% of requests). ChatGPT spends 34.82% and Claude 34.16% of fetches on 404s vs Googlebot's 8.22%. Establishes that (a) what an AI crawler ingests is the raw HTML byte stream with no CSS/JS applied, and (b) per-fetch yield is already terrible, so wasted tokens per fetch compound."
    },
    {
      "id": "vercel-ai-crawler-rendering",
      "title": "The rise of the AI crawler",
      "url": "https://vercel.com/blog/the-rise-of-the-ai-crawler",
      "type": "dataset",
      "publisher": "Vercel (with MERJ)",
      "verified": "2026-08-21",
      "keyFindings": "Log analysis across Vercel's network, nextjs.org and two job boards: none of the major AI crawlers (GPTBot, ClaudeBot, PerplexityBot) execute JavaScript; they fetch JS files as text (ChatGPT 11.50%, Claude 23.84% of requests) without executing. Consequence for this domain: JSON-LD injected client-side (GTM, React hydration) is invisible to AI crawlers even though Googlebot renders it."
    },
    {
      "id": "vercel-ai-crawler-study",
      "title": "The rise of the AI crawler",
      "url": "https://vercel.com/blog/the-rise-of-the-ai-crawler",
      "type": "dataset",
      "publisher": "Vercel",
      "verified": "2026-08-21",
      "keyFindings": "Empirical crawler-log study. 'none of the major AI crawlers currently render JavaScript. This includes: OpenAI (OAI-SearchBot, ChatGPT-User, GPTBot) Anthropic (ClaudeBot)'. They fetch JS without executing it (ChatGPT 11.50%, Claude 23.84% of requests). Content-type mix: ChatGPT HTML 57.70%; Claude images 35.17%; Googlebot HTML 31.00%, JSON 29.34%, plain text 20.77%, JS 15.25%. Implication: server-rendered semantic HTML is what these crawlers actually see, and Claude's crawler does fetch image bytes while ChatGPT's is far more HTML-weighted."
    },
    {
      "id": "vercel-ai-crawlers",
      "title": "The rise of the AI crawler",
      "url": "https://vercel.com/blog/the-rise-of-the-ai-crawler",
      "publisher": "Vercel",
      "type": "study",
      "keyFindings": "\"none of the major AI crawlers currently render JavaScript.\" ChatGPT fetches JS files in 11.50% of requests and Claude 23.84%, but neither executes them. Only Gemini renders, by leveraging Googlebot's infrastructure. Preconnect, a render-pipeline hint, cannot be acted on by non-rendering crawlers.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "vercel-crawlers",
      "title": "The rise of the AI crawler (Vercel / Merj log study)",
      "url": "https://vercel.com/blog/the-rise-of-the-ai-crawler",
      "type": "study",
      "publisher": "Vercel",
      "verified": "2026-08-21",
      "keyFindings": "None of GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Meta, ByteDance or Perplexity crawlers execute JavaScript; Gemini rides Googlebot infra (renders) and AppleBot renders. ChatGPT spends 11.50% of requests on JS files, Claude 23.84% — fetched as text, never executed. Crawl waste: ChatGPT 34.82% of fetches hit 404s and 14.36% follow redirects; Claude 34.16% hit 404s; Googlebot only 8.22%/1.49%."
    },
    {
      "id": "vercel-merj-ai-crawler-study",
      "title": "The rise of the AI crawler",
      "url": "https://vercel.com/blog/the-rise-of-the-ai-crawler",
      "type": "study",
      "publisher": "Vercel (with MERJ)",
      "verified": "2026-08-21",
      "keyFindings": "Server-log study across the Vercel network and nextjs.org. \"None of the major AI crawlers currently render JavaScript. This includes: OpenAI (OAI-SearchBot, ChatGPT-User, GPTBot), Anthropic (ClaudeBot) ... Perplexity (PerplexityBot).\" Exception: \"AppleBot renders JavaScript through a browser-based crawler, similar to Googlebot.\" ChatGPT fetches are 57.70% HTML; Claude 23.84% JavaScript, 35.17% images. Volumes cited: GPTBot 569M requests/month, Claude 370M; the four bots together ~1.3B fetches, ~28% of Googlebot volume. Load-bearing consequence for this domain: any head element injected client-side is invisible to every major AI crawler except Applebot."
    },
    {
      "id": "vercel-rise-of-ai-crawler",
      "title": "The rise of the AI crawler",
      "url": "https://vercel.com/blog/the-rise-of-the-ai-crawler",
      "type": "study",
      "publisher": "Vercel / MERJ",
      "verified": "2026-08-21",
      "keyFindings": "Published 2024-12-17 (Zecchini, Moore, Ubl, Siddle). Measured over one month on Vercel's network: GPTBot 569M requests, Claude 370M, together ~1/5 of Googlebot's 4.5B. Core finding: none of the major AI crawlers (OpenAI, Anthropic, Meta, ByteDance, Perplexity) render JavaScript, though ChatGPT fetches JS files in 11.50% of requests and Claude 23.84% without executing them. Content mix: ChatGPT 57.70% HTML; Claude 35.17% images. Waste: ChatGPT 34.82% and Claude 34.16% of fetches hit 404s vs Googlebot's 8.22%."
    },
    {
      "id": "w3c-accname",
      "title": "Accessible Name and Description Computation 1.1",
      "url": "https://www.w3.org/TR/accname/",
      "type": "spec",
      "publisher": "W3C",
      "verified": "2026-08-20",
      "keyFindings": "W3C Recommendation, 18 December 2018 — a ratified standard. Defines the precedence order that turns markup into the accessible name agents actually read: aria-labelledby > aria-label > native host-language attributes/elements (HTML alt, title, <label>; SVG <desc>) > embedded control values > element content (for roles allowing name-from-content) > tooltip attributes. This is the causal bridge from alt/label/aria-* to the strings in a Playwright/Anthropic/Chrome-DevTools snapshot."
    },
    {
      "id": "w3c-accname-11",
      "title": "Accessible Name and Description Computation 1.1",
      "url": "https://www.w3.org/TR/accname-1.1",
      "type": "spec",
      "publisher": "W3C",
      "verified": "2026-08-21",
      "keyFindings": "W3C Recommendation, 18 December 2018. Governs the idref-based `aria-labelledby` and `aria-describedby` computation, and ranks `alt` below `aria-labelledby` and `aria-label`."
    },
    {
      "id": "w3c-accname-12",
      "title": "Accessible Name and Description Computation 1.2",
      "url": "https://www.w3.org/TR/accname-1.2",
      "type": "spec",
      "publisher": "W3C",
      "verified": "2026-08-21",
      "keyFindings": "Defines the LabelledBy step: \"For each IDREF: Set the current node to the node referenced by the IDREF… Compute the text alternative of the current node\" — so a duplicated id silently renames a control."
    },
    {
      "id": "w3c-act-rules-format",
      "title": "Accessibility Conformance Testing (ACT) Rules Format 1.1",
      "url": "https://www.w3.org/TR/act-rules-format/",
      "type": "spec",
      "publisher": "W3C",
      "verified": "2026-08-20",
      "keyFindings": "A W3C-published format for writing test rules. A conforming rule 'must consist of at least' Descriptive Title, Rule Identifier, Rule Description, Rule Type, Accessibility Requirements Mapping, Rule Input, Applicability, Expectations, a Background section containing Assumptions and Accessibility Support, passed/failed/inapplicable Examples, Rule Versions (change history), Format Version, and Glossary. Two mandated fields have no counterpart anywhere in AI-readiness tooling: Assumptions (state what must be true for the rule to be valid) and Accessibility Support (document which consumers actually support the technique). Our 'mechanism' and 'consumers' fields are the AI-readiness translation of exactly those two."
    },
    {
      "id": "w3c-aria-12-hidden",
      "title": "WAI-ARIA 1.2 — aria-hidden",
      "url": "https://www.w3.org/TR/wai-aria-1.2/#aria-hidden",
      "type": "spec",
      "publisher": "W3C",
      "verified": "2026-08-21",
      "keyFindings": "W3C Recommendation, 06 June 2023. §7.1 \"Excluding Elements from the Accessibility Tree\" specifies that `aria-hidden=\"true\"` removes the element and its descendants from the tree."
    },
    {
      "id": "w3c-aria-12-states",
      "title": "WAI-ARIA 1.2 — states and properties",
      "url": "https://www.w3.org/TR/wai-aria-1.2/#state_prop_def",
      "type": "spec",
      "publisher": "W3C",
      "verified": "2026-08-21",
      "keyFindings": "W3C Recommendation, 06 June 2023. Defines each state and property's allowed value type, and that user agents expose the default when a state is undefined for the role."
    },
    {
      "id": "w3c-html-aam",
      "title": "HTML Accessibility API Mappings 1.0",
      "url": "https://www.w3.org/TR/html-aam-1.0/",
      "type": "draft-spec",
      "publisher": "W3C",
      "verified": "2026-08-21",
      "keyFindings": "W3C Working Draft (5 August 2026). The normative element→role table: main→main, nav→navigation, header (body-scoped)→banner, footer (body-scoped)→contentinfo, article→article, aside→complementary, section (named)→region, figure→figure, figcaption→caption, table→table, th→columnheader/rowheader, h1-h6→heading with aria-level, img[alt]→image, img[alt='']→none/presentation, label→provides accessible naming for form controls. Critically for the <time> signal: time→GENERIC role, i.e. <time datetime> is invisible in the accessibility tree."
    },
    {
      "id": "w3c-i18n-lang-declarations",
      "title": "W3C i18n — declaring language in HTML",
      "url": "https://www.w3.org/International/questions/qa-html-language-declarations",
      "type": "spec",
      "publisher": "W3C",
      "verified": "2026-08-21",
      "keyFindings": "\"You should always declare the language of the text in a page using a language attribute on the `html` tag.\""
    },
    {
      "id": "w3c-json-ld-11",
      "title": "JSON-LD 1.1 — A JSON-based Serialization for Linked Data",
      "url": "https://www.w3.org/TR/json-ld11/",
      "type": "spec",
      "publisher": "W3C",
      "verified": "2026-08-21",
      "keyFindings": "W3C Recommendation, 16 July 2020. Ratified standard status for the serialization format underlying every signal in this domain (the vocabulary, schema.org, is a separate community-maintained vocabulary, not a W3C Rec)."
    },
    {
      "id": "w3c-jsonld-api",
      "title": "JSON-LD 1.1 Processing Algorithms and API",
      "url": "https://www.w3.org/TR/json-ld11-api",
      "type": "spec",
      "publisher": "W3C",
      "verified": "2026-08-21",
      "keyFindings": "W3C Recommendation, 16 July 2020. Defines the Expansion Algorithm that resolves every entry key against the active context — the normative step schema validation depends on."
    },
    {
      "id": "w3c-tdmrep-cg-final",
      "title": "TDM Reservation Protocol (TDMRep) — Final Community Group Report",
      "url": "https://www.w3.org/community/reports/tdmrep/CG-FINAL-tdmrep-20240202/",
      "type": "draft-spec",
      "publisher": "W3C Text and Data Mining Reservation Protocol Community Group",
      "verified": "2026-08-20",
      "keyFindings": "Three delivery methods: /.well-known/tdmrep.json, `tdm-reservation` / `tdm-policy` HTTP response headers, and HTML <meta> tags. tdm-reservation is boolean 1/0; tdm-policy points at an ODRL 2.2 policy document. Status is a Final COMMUNITY GROUP Report (Feb 2024) under the W3C Community Final Specification Agreement — NOT a W3C Recommendation, no Standards Track standing. The spec itself names essentially one implementer (an 'swpawning.ai' API); no AI model provider is listed as a consumer."
    },
    {
      "id": "w3c-tdmrep-final-report",
      "title": "TDM Reservation Protocol (TDMRep) — Final Community Group Report",
      "url": "https://www.w3.org/community/reports/tdmrep/CG-FINAL-tdmrep-20240202/",
      "type": "draft-spec",
      "publisher": "W3C Text and Data Mining Reservation Protocol Community Group (2 Feb 2024)",
      "verified": "2026-08-20",
      "keyFindings": "The only quasi-standard that defines AI/TDM opt-out via HTML head metadata. Defines <meta name=\"tdm-reservation\" content=\"1|0\"> and <meta name=\"tdm-policy\" content=\"<URL>\">, plus a /.well-known/tdmrep.json file and HTTP response header. Explicitly \"not a W3C Standard\" — a Final Community Group Report. Framed as a technical answer to EU DSM Directive Article 4. Contains no statements about implementing consumers or adoption."
    },
    {
      "id": "w3c-wai-aria-1-2",
      "title": "Accessible Rich Internet Applications (WAI-ARIA) 1.2",
      "url": "https://www.w3.org/TR/wai-aria-1.2/",
      "type": "spec",
      "publisher": "W3C",
      "verified": "2026-08-20",
      "keyFindings": "W3C Recommendation, 6 June 2023. Defines the eight landmark roles — banner, complementary, contentinfo, form, main, navigation, region, search — as 'regions of the page intended as navigational landmarks'. These are the role names that appear verbatim in agent snapshots and in Readability's UNLIKELY_ROLES deletion list."
    },
    {
      "id": "w3c-webmcp-spec",
      "title": "WebMCP — Draft Community Group Report",
      "url": "https://webmachinelearning.github.io/webmcp/",
      "type": "draft-spec",
      "publisher": "W3C Web Machine Learning Community Group",
      "verified": "2026-08-21",
      "keyFindings": "Draft Community Group Report dated 19 August 2026. Explicitly 'not a W3C Standard nor is it on the W3C Standards Track.' Defines the `ModelContext` interface on `document.modelContext` (moved from `navigator.modelContext`) with `registerTool()`, `getTools()`, `executeTool()`; `ModelContextTool` carries `name`, `description`, `inputSchema` (JSON Schema) and optional `ToolAnnotations` (readOnlyHint, untrustedContentHint). Permissions-Policy feature name is 'tools'. The spec defines NO manifest format; the declarative HTML-form API (§4.3) is marked 'entirely a TODO'."
    },
    {
      "id": "w3c-webml-wg",
      "title": "W3C Web Machine Learning Working Group",
      "url": "https://www.w3.org/groups/wg/webmachinelearning/",
      "type": "spec",
      "publisher": "W3C",
      "verified": "2026-08-20",
      "keyFindings": "WG mission is 'to develop APIs for enabling efficient machine learning inference in the browser'; chartered until 30 April 2027. WebMCP is NOT a Working Group deliverable — it lives in the Web Machine Learning COMMUNITY Group, which has no W3C Recommendation track authority."
    },
    {
      "id": "wasp",
      "title": "WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks",
      "url": "https://arxiv.org/abs/2504.18575",
      "type": "study",
      "publisher": "arXiv (Meta / UCL)",
      "verified": "2026-08-20",
      "keyFindings": "Low-effort, human-written injections embedded in realistic web pages partially succeed in up to 86% of cases against frontier models. Full attacker-goal completion is lower, which the authors call 'security by incompetence' — meaning the exposure is not a model-quality problem that will self-correct."
    },
    {
      "id": "wcag-22",
      "title": "WCAG 2.2",
      "url": "https://www.w3.org/TR/WCAG22",
      "type": "spec",
      "publisher": "W3C",
      "verified": "2026-08-21",
      "keyFindings": "W3C Recommendation of 12 December 2024. SC 3.1.1 (Level A): \"The natural language of each web page can be programmatically determined.\""
    },
    {
      "id": "wcag-f41",
      "title": "WCAG 2.2 — failure F41",
      "url": "https://www.w3.org/WAI/WCAG22/Techniques/failures/F41",
      "type": "spec",
      "publisher": "W3C",
      "verified": "2026-08-21",
      "keyFindings": "The failure and technique set behind SC 2.4.2 Page Titled (Level A), which is the page-title expectation this audit checks."
    },
    {
      "id": "wcag-g1",
      "title": "WCAG Technique G1: Adding a link at the top of each page that goes directly to the main content area",
      "url": "https://www.w3.org/WAI/WCAG21/Techniques/general/G1",
      "publisher": "W3C WAI",
      "type": "spec",
      "keyFindings": "The audit's own cited source is a human keyboard/screen-reader technique. It frames the benefit entirely in terms of users who navigate sequentially; it makes no claim about automated agents or crawlers.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "wcag-language-of-page",
      "title": "Understanding SC 3.1.1: Language of Page (WCAG 2.2)",
      "url": "https://www.w3.org/WAI/WCAG22/Understanding/language-of-page.html",
      "type": "spec",
      "publisher": "W3C Web Accessibility Initiative",
      "verified": "2026-08-21",
      "keyFindings": "Level A success criterion: \"The default human language of each web page can be programmatically determined.\" Named consumers of the lang attribute: screen readers (pronunciation rules), visual browsers (character/script rendering), media players (captions), text-to-speech. All human-assistive; no AI-crawler consumer is named. This is the ratified-standard leg supporting html lang — the AI leg is missing."
    },
    {
      "id": "weaviate-catalog",
      "title": "Weaviate /.well-known/ai-catalog.json (live)",
      "url": "https://weaviate.io/.well-known/ai-catalog.json",
      "publisher": "Weaviate",
      "type": "vendor-doc",
      "keyFindings": "Live manifest, specVersion 1.0, host object with displayName/docs/logo, 9 entries (docs, agent skills, OpenAPI, sitemap) with representativeQueries and tags.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "weaviate-catalog-urls",
      "title": "Weaviate live manifest entries",
      "url": "https://weaviate.io/.well-known/ai-catalog.json",
      "publisher": "Weaviate",
      "type": "vendor-doc",
      "keyFindings": "9 entries: docs guides, agent skills, REST/OpenAPI spec, sitemap — each a dereferenceable URL.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "web-almanac-2024-accessibility",
      "title": "Web Almanac 2024 — Accessibility chapter",
      "url": "https://almanac.httparchive.org/en/2024/accessibility",
      "type": "dataset",
      "publisher": "HTTP Archive",
      "verified": "2026-08-20",
      "keyFindings": "Prior-year baseline for trend lines: 69% of images pass the alt audit (up from 59% in 2022), 30% of alt attributes are empty (legitimate for decorative images) and 15% of images have no alt at all; 57% of mobile sites keep proper heading hierarchy; <main> 37%; 43% use either native landmarks or ARIA roles, up from 35% in 2021; 32% of inputs named by <label>; 5.5% of sites with tables use captions; 24% of pages likely have skip links."
    },
    {
      "id": "web-almanac-2025-accessibility",
      "title": "Web Almanac 2025 — Accessibility chapter",
      "url": "https://almanac.httparchive.org/en/2025/accessibility",
      "type": "dataset",
      "publisher": "HTTP Archive",
      "verified": "2026-08-20",
      "keyFindings": "Adoption baseline over millions of pages. Alt text: 69% of images pass the Lighthouse audit; ~8.5% of alt values end in a file extension (.jpg/.png), i.e. worthless strings. Heading order: 59% of mobile sites pass the ordered-headings audit (+~2pp vs 2024). Landmarks: <main> 40.72% (47.34% incl. role), <header> 65.99%, <nav> 67.73%, <footer> 66.38%. Forms: ~35% of mobile inputs get their accessible name from <label>. Tables: 1.6% of desktop sites use <caption>; 4.9% of mobile tables use role='presentation'. Skip links: ~24% of desktop and mobile pages."
    },
    {
      "id": "web-dev-agent-friendly-sites",
      "title": "Build agent-friendly websites",
      "url": "https://web.dev/articles/ai-agent-site-ux",
      "type": "vendor-doc",
      "publisher": "Google / web.dev",
      "verified": "2026-08-20",
      "keyFindings": "Does not mention schema.org, structured data or JSON-LD at all. Names three agent-facing representations instead: screenshots, raw HTML/DOM, and the accessibility tree (\"a browser-native API [that] distills the DOM into what's most important: roles, names, and states of interactive elements\"). Strong counter-evidence that Google's own agent guidance treats schema.org as irrelevant to browser agents."
    },
    {
      "id": "webalmanac-2024-structured-data",
      "title": "Web Almanac 2024 — Structured Data chapter",
      "url": "https://almanac.httparchive.org/en/2024/structured-data",
      "type": "dataset",
      "publisher": "HTTP Archive",
      "verified": "2026-08-21",
      "keyFindings": "Format adoption by page: RDFa 66% (inflated — Open Graph prefixes are parsed as RDFa), Open Graph 64%, Twitter meta 45%, JSON-LD 41% (up from 34% in 2022, fastest-growing), Microdata 26%. Most common JSON-LD types (mobile): WebSite 12.73%, Organization 7.16%, BreadcrumbList 5.66%, LocalBusiness 3.97%, ItemList 2.44%. Notes a shift in structured data's role from pure SEO toward AI/semantic metadata."
    },
    {
      "id": "webalmanac-2025-generative-ai",
      "title": "Web Almanac 2025 — Generative AI chapter",
      "url": "https://almanac.httparchive.org/en/2025/generative-ai",
      "type": "dataset",
      "publisher": "HTTP Archive",
      "verified": "2026-08-20",
      "keyFindings": "July 2025 HTTP Archive crawl, ~12.9M sites. llms.txt present with valid entries on 2.13% desktop / 2.1% mobile (97.87% / 97.9% absent). robots.txt present with directives on 94.1% of sites. 'gptbot' directive grew from 2.6% (2024) to ~4.5% (2025). Most-targeted AI user agents: GPTBot, ClaudeBot, Google-Extended, ChatGPT-User, PerplexityBot; adoption rises sharply with site rank. .ai TLD growth: top-1k 0→2, top-10k 1→21, top-1M 273→1,606, top-10M 2,824→11,848 (2022→2025). No adoption metrics available for WebMCP or other agent protocols."
    },
    {
      "id": "webalmanac-2025-seo",
      "title": "Web Almanac 2025 — SEO chapter",
      "url": "https://almanac.httparchive.org/en/2025/seo",
      "type": "dataset",
      "publisher": "HTTP Archive",
      "verified": "2026-08-20",
      "keyFindings": "50% of all pages contain structured data. AI-bot robots.txt directives: gptbot 4.5% desktop / 4.2% mobile (up ~55% from 2.9% / 2.7% in 2024); claudebot 1.9%→3.6% desktop; ccbot 2.7%→3.5%; petalbot, amazonbot, google-extended entering multi-percent range. llms.txt adoption ~2%, and critically 39.6% of llms.txt files are generated by the All in One SEO plugin — i.e. a large share are CMS defaults rather than deliberate choices. VideoObject markup only 0.9% of pages."
    },
    {
      "id": "webarena-paper",
      "title": "WebArena: A Realistic Web Environment for Building Autonomous Agents",
      "url": "https://arxiv.org/abs/2307.13854",
      "type": "study",
      "publisher": "arXiv (Zhou et al., 2023)",
      "verified": "2026-08-20",
      "keyFindings": "The standard web-agent benchmark. Its environment renders page content in multiple modes — raw HTML (DOM tree), screenshots, and accessibility trees — where the accessibility tree is a subset of the DOM containing role, text content and properties per element. Note honestly: the abstract itself does not discuss observation modalities; the a11y-tree observation is a property of the released environment and config, not a headline claim of the paper."
    },
    {
      "id": "webarena-repo",
      "title": "web-arena-x/webarena repository",
      "url": "https://github.com/web-arena-x/webarena",
      "type": "dataset",
      "publisher": "WebArena / CMU",
      "verified": "2026-08-20",
      "keyFindings": "Config-level confirmation: observation_type=\"accessibility_tree\" is a first-class setting, and the README describes getting 'the text observation (e.g., html, accessibility tree) through obs[\"text\"]'. The README does not argue for why the a11y tree is preferred, so this is adoption evidence rather than a justification."
    },
    {
      "id": "webdatacommons-2024-stats",
      "title": "Web Data Commons Extraction Report — October 2024 Common Crawl Corpus",
      "url": "https://webdatacommons.org/structureddata/2024-12/stats/stats.html",
      "type": "dataset",
      "publisher": "Web Data Commons (University of Mannheim)",
      "verified": "2026-08-21",
      "keyFindings": "2.39B HTML pages parsed. Structured data on 1.3B pages (51.25%) from 16.5M pay-level domains (44.12%). By format: JSON-LD 11.6M domains / 834M URLs / 48B triples; Microdata 7.6M domains; hCard 3.5M; RDFa 475K domains. Top JSON-LD classes by domain: WebSite 8.5M, ImageObject 6.9M, Organization 6.6M, SearchAction 6.6M, WebPage 6.3M. By entity count: ListItem 1.7B, ImageObject 1B, Organization 717M, Offer 616M, BreadcrumbList 585M. Establishes adoption baselines for Organization, Offer, BreadcrumbList and (notably) SearchAction."
    },
    {
      "id": "webkit-standards-position-webmcp",
      "title": "WebKit standards-positions — WebMCP (issue 670)",
      "url": "https://github.com/WebKit/standards-positions/issues/670",
      "type": "vendor-doc",
      "publisher": "Apple / WebKit",
      "verified": "2026-08-20",
      "keyFindings": "Opened 2026-05-28, closed 2026-06-17. Labels include 'position: oppose' plus concern labels: duplication, internationalization, privacy, security, venue, use cases, portability, API design, meaningful user consent. Apple formally OPPOSES WebMCP."
    },
    {
      "id": "webmcp-alternatives-rejected-manifest",
      "title": "WebMCP explainer — Alternatives Considered §2: Static Declarative Manifests",
      "url": "https://raw.githubusercontent.com/webmachinelearning/webmcp/main/README.md",
      "publisher": "W3C Web Machine Learning Community Group",
      "type": "spec",
      "keyFindings": "Static manifest files were considered and rejected: \"Static manifests prevent web developers from dynamically adding, updating, or removing tools based on the active page state or user authentication status\" and \"Manifests cannot contain executable code, meaning developers would still need an imperative way to register execution handlers.\" Grep of README.md, declarative-api-explainer.md, index.bs and implementation-status.md finds zero occurrences of \"/.well-known\".",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "webmcp-declarative-explainer",
      "title": "WebMCP Declarative API Explainer",
      "url": "https://raw.githubusercontent.com/webmachinelearning/webmcp/main/declarative-api-explainer.md",
      "publisher": "W3C Web Machine Learning Community Group",
      "type": "spec",
      "keyFindings": "Defines toolname, tooldescription, toolautosubmit on <form> and toolparamdescription on form-associated controls. Example: <form toolname=\"search-cars\" tooldescription=\"Perform a car make/model search\"> with toolparamdescription on inputs. Targets <form> and HTML form-associated elements.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "webmcp-impl-status",
      "title": "WebMCP implementation-status.md",
      "url": "https://raw.githubusercontent.com/webmachinelearning/webmcp/main/implementation-status.md",
      "publisher": "W3C Web Machine Learning Community Group",
      "type": "spec",
      "keyFindings": "Named consumers: Brave — \"Experimental support is added to Leo AI chat\"; Chrome — Origin Trial live in Chrome 149; Edge — Origin Trial live in Edge 150. Firefox and Safari still in standards discussion.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "webmcp-implementation-status",
      "title": "WebMCP — Implementation Status",
      "url": "https://raw.githubusercontent.com/webmachinelearning/webmcp/main/implementation-status.md",
      "type": "draft-spec",
      "publisher": "W3C Web Machine Learning CG",
      "verified": "2026-08-20",
      "keyFindings": "Brave: 'Experimental support is added to Leo AI chat' (brave-browser issue 55232). Chrome: 'An Origin Trial is live in Chrome 149.' Edge: 'An Origin Trial is live in Edge 150.' Firefox: standards-positions #1412 + Bugzilla 2018306. Safari: WebKit standards-positions #670. Brave Leo is the only named shipping agent consumer."
    },
    {
      "id": "webmcp-link-451",
      "title": "webmcp.link (docsUrl cited by the audit)",
      "url": "https://webmcp.link/",
      "publisher": "third party",
      "type": "article",
      "keyFindings": "(Returns HTTP 451 — the legal-block status IS the observation.) Returns HTTP 451 Unavailable For Legal Reasons — the docsUrl both WebMCP audits point at is dead. Should be replaced with developer.chrome.com/docs/ai/webmcp.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "webmcp-spec-draft",
      "title": "Web Model Context API (WebMCP) draft specification",
      "url": "https://webmachinelearning.github.io/webmcp/",
      "publisher": "W3C Web Machine Learning Community Group",
      "type": "spec",
      "keyFindings": "Section 4.3 \"Declarative WebMCP\" exists but is marked \"This section is entirely a TODO. For now, refer to the explainer draft.\" Defines a \"synthesize a declarative JSON Schema object\" algorithm over <form> and form-associated inputs. No /.well-known manifest is referenced anywhere in the spec.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "webmcp-spec-draft-nomanifest",
      "title": "Web Model Context API (WebMCP) draft specification",
      "url": "https://webmachinelearning.github.io/webmcp/",
      "publisher": "W3C Web Machine Learning Community Group",
      "type": "spec",
      "keyFindings": "Confirmed on fetch: no reference to a .well-known manifest file anywhere in the specification. Tool discovery is via document.modelContext / getTools() in the page, plus browser synthesis from annotated forms.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "webmcp-spec-no-nav",
      "title": "Web Model Context API (WebMCP) draft specification",
      "url": "https://webmachinelearning.github.io/webmcp/",
      "publisher": "W3C Web Machine Learning Community Group",
      "type": "spec",
      "keyFindings": "Checked as an adjacent agent-facing standard: defines in-page tool registration and form-synthesized tools only. No site-navigation manifest, no .well-known or root JSON discovery file of any kind.",
      "verified": "2026-08-21",
      "accessed": "2026-08-21"
    },
    {
      "id": "webstatus-declarative-webmcp",
      "title": "Baseline / web-features API: declarative-webmcp — \"Form-associated WebMCP attributes\"",
      "url": "https://api.webstatus.dev/v1/features?q=webmcp",
      "publisher": "Google / WebDX Community Group",
      "type": "vendor-doc",
      "keyFindings": "Named web feature `declarative-webmcp` = \"Form-associated WebMCP attributes\", spec https://webmachinelearning.github.io/webmcp/. WPT: chrome stable 0.6, chrome experimental 1.0, chrome_android experimental 0.867, edge experimental 0.6, firefox 0, safari 0. Chrome daily usage 0.00000274. Vendor positions: Apple oppose (WebKit standards-positions #670), Mozilla neutral (#1412). Baseline status: limited. Sibling feature document-modelcontext (imperative API) usage 0.00722698.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "websub",
      "title": "WebSub (W3C Recommendation)",
      "url": "https://www.w3.org/TR/websub/",
      "type": "spec",
      "publisher": "W3C",
      "verified": "2026-08-20",
      "keyFindings": "W3C Recommendation. Publishers 'MUST implement at least one' of Link Headers or embedded link elements, advertising at least one rel=hub and exactly one rel=self (the canonical topic URL). Discovery checks Link headers first, then embedded link elements; for HTML, link elements are recommended in <head> only."
    },
    {
      "id": "wellknownmcp-org",
      "title": "wellknownmcp.org",
      "url": "https://wellknownmcp.org/",
      "type": "article",
      "publisher": "wellknownmcp org (community)",
      "verified": "2026-08-20",
      "keyFindings": "Self-describes as 'a working collection of doctrine, tools and reference implementations' — explicitly a community/third-party proposal, NOT an official MCP standard. Advocates llms.txt, .well-known checks, AGENTS.md, security.txt and RFC 9728. Provides NO named consumers and NO adoption metrics for its own conventions."
    },
    {
      "id": "what-gets-cited",
      "title": "What Gets Cited: Competitive GEO in AI Answer Engines",
      "url": "https://arxiv.org/abs/2605.25517",
      "publisher": "arXiv",
      "type": "study",
      "verified": "2026-08-20",
      "keyFindings": "252,000-trial controlled RAG testbed, 18 factors, six LLMs. 'Weaker Social Proof' OR 2.14 to >10,000, significant in 4/6 models; 'Claims With Evidence' OR 2.09 to >10,000, significant in 5/6; 'Overly Promotional' significant in only 3/6 with mixed direction and 'insufficient to establish clear guideline'. Social proof is operationally defined as 'Fewer or lower ratings/reviews'. Gatekeepers (topic match, price, recency, list position) all OR >>10,000 dwarf trust cues. Practical guidance: topic terms early, price and specs, comparisons, current dates, evidence-backed claims replacing hedging.",
      "accessed": "2026-08-21"
    },
    {
      "id": "whatwg-address",
      "title": "WHATWG HTML — the address element",
      "url": "https://html.spec.whatwg.org/multipage/sections.html",
      "publisher": "WHATWG",
      "type": "spec",
      "keyFindings": "'The address element represents the contact information for its nearest article or body element ancestor.' and 'The address element must not be used to represent arbitrary addresses (e.g. postal addresses), unless those addresses are in fact the relevant contact information.' General mailing addresses should use <p>. The audit's guidance and code example ignore this restriction.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "whatwg-aside",
      "title": "WHATWG HTML — 4.3 Sections (aside, address)",
      "url": "https://html.spec.whatwg.org/multipage/sections.html",
      "publisher": "WHATWG",
      "type": "spec",
      "keyFindings": "'The aside element represents a section of a page that consists of content that is tangentially related to the content around the aside element, and which could be considered separate from that content' — typically sidebars, pull quotes, advertising, navigation groups. The spec definition matches exactly what the extraction libraries strip.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "whatwg-code-element",
      "title": "HTML Standard — the code element",
      "url": "https://html.spec.whatwg.org/multipage/text-level-semantics.html#the-code-element",
      "type": "spec",
      "publisher": "WHATWG",
      "verified": "2026-08-21",
      "keyFindings": "§4.5.15: \"There is no formal way to indicate the language of computer code being marked up.\" The `language-*` class is a convention, not a specification."
    },
    {
      "id": "whatwg-declarative-refresh",
      "title": "HTML Standard — shared declarative refresh steps",
      "url": "https://html.spec.whatwg.org/multipage/document-lifecycle.html#shared-declarative-refresh-steps",
      "type": "spec",
      "publisher": "WHATWG",
      "verified": "2026-08-21",
      "keyFindings": "Defines the `refresh` pragma and the shared declarative refresh steps that navigate or reload the document."
    },
    {
      "id": "whatwg-html-dfn",
      "title": "HTML Standard — The dfn element",
      "url": "https://html.spec.whatwg.org/multipage/text-level-semantics.html#the-dfn-element",
      "type": "spec",
      "publisher": "WHATWG",
      "verified": "2026-08-21",
      "keyFindings": "Ratified living standard providing a semantic definition primitive: 'The dfn element represents the defining instance of a term.' 'The paragraph, description list group, or section that is the nearest ancestor of the dfn element must also contain the definition(s) for the term given by the dfn element.' Term resolution via title attribute, child abbr title, or descendant text. 'An a element that links to a dfn element represents an instance of the term defined by the dfn element.' Counter-evidence: no answer engine documents consuming dfn for definition extraction."
    },
    {
      "id": "whatwg-html-metadata-names",
      "title": "HTML Standard — 4.2.5.1 Standard metadata names",
      "url": "https://html.spec.whatwg.org/multipage/semantics.html",
      "type": "spec",
      "publisher": "WHATWG",
      "verified": "2026-08-20",
      "keyFindings": "Defines author (\"a free-form string giving the name of one of the page's authors\"), description (\"must be appropriate for use in a directory of pages, e.g. in a search engine\"), keywords (\"a set of comma-separated tokens\"), generator. Notably, 'robots' is NOT a standard metadata name in the HTML spec — it is a de-facto convention registered outside the spec. No consumer obligations are specified for any of these names."
    },
    {
      "id": "whatwg-links",
      "title": "HTML Standard — 4.6.8 Link types (next, prev)",
      "url": "https://html.spec.whatwg.org/multipage/links.html#sec-link-types",
      "publisher": "WHATWG",
      "type": "spec",
      "verified": "2026-08-20",
      "keyFindings": "Both 'next' and 'prev' are defined as Hyperlink annotations and the link-types conformance table marks the <link> column as 'not allowed' for both. Valid on <a>, <area> and <form> only. The audit checks head <link> elements and recommends <link rel=\"prev\"> markup, which the living standard forbids.",
      "accessed": "2026-08-21"
    },
    {
      "id": "whatwg-metadata-names",
      "title": "HTML Standard — standard metadata names",
      "url": "https://html.spec.whatwg.org/multipage/semantics.html#standard-metadata-names",
      "type": "spec",
      "publisher": "WHATWG",
      "verified": "2026-08-21",
      "keyFindings": "Defines `author` as a standard metadata name: \"The value must be a free-form string giving the name of one of the page's authors.\" Assigns no consumer behaviour to it."
    },
    {
      "id": "whatwg-tabindex",
      "title": "HTML Standard — the tabindex attribute",
      "url": "https://html.spec.whatwg.org/multipage/interaction.html#the-tabindex-attribute",
      "type": "spec",
      "publisher": "WHATWG",
      "verified": "2026-08-21",
      "keyFindings": "A positive `tabindex` places the element \"in the tabindex-ordered focus navigation scope\" ordered by its numeric value, with the advisory \"Developers should use caution when using values other than 0 or −1\"."
    },
    {
      "id": "whatwg-tables",
      "title": "HTML Standard — tabular data",
      "url": "https://html.spec.whatwg.org/multipage/tables.html",
      "type": "spec",
      "publisher": "WHATWG",
      "keyFindings": "Defines `colspan`, `rowspan`, `thead`/`tbody`/`tfoot` and the header-association algorithm — the table structure that has no markdown equivalent.",
      "verified": "2026-08-20"
    },
    {
      "id": "whatwg-time-element",
      "title": "HTML Standard — the time element",
      "url": "https://html.spec.whatwg.org/multipage/text-level-semantics.html#the-time-element",
      "type": "spec",
      "publisher": "WHATWG",
      "verified": "2026-08-20",
      "keyFindings": "Living Standard. 'The time element represents its contents, along with a machine-readable form of those contents in the datetime attribute.' 'If present, its value must be a representation of the element's contents in a machine-readable format.' COUNTER-EVIDENCE: the spec defines parsing syntaxes but prescribes NO user-agent behaviour for consuming the value — there is no standardised consumer."
    },
    {
      "id": "wild-card-agents-json-repo",
      "title": "wild-card-ai/agents-json — open specification for API/agent contracts",
      "url": "https://github.com/wild-card-ai/agents-json",
      "publisher": "Wild Card AI",
      "type": "repo",
      "keyFindings": "Spec v0.1.0 built on OpenAPI; 1,314 stars; explicitly proposes placement at /.well-known/agents.json 'so it is easily discoverable by agents accessing web services'; README names NO vendor or agent that consumes the format (Stripe/Resend/Google Sheets are self-authored demos). GitHub API: archived=false but pushed_at=2025-08-21T22:17:46Z (dormant 12 months), 12 open issues, homepage listed as https://agents-json.com. Last three commits are housekeeping only.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "wildcard-agents-json-repo",
      "title": "wild-card-ai/agents-json",
      "url": "https://github.com/wild-card-ai/agents-json",
      "type": "draft-spec",
      "publisher": "Wildcard AI",
      "verified": "2026-08-20",
      "keyFindings": "1,314 stars, 66 forks, created 2025-01-30, LAST PUSH 2025-08-21 — twelve months stale as of 2026-08-20. Repo description is empty. Homepage `https://agents-json.com` FAILS TO RESOLVE (curl exit 6/000). Documentation host docs.wild-card.ai serves an EXPIRED TLS certificate (notBefore 2026-01-09, notAfter 2026-04-09). Spec version stuck at 0.1.0."
    },
    {
      "id": "wislr-48day-log-study",
      "title": "AI Bot Traffic Is Accelerating Fast. 48 Days of Server Logs Expose What GPTBot, ChatGPT, ClaudeBot, and 16 Others Are Doing",
      "url": "https://www.wislr.com/articles/ai-bot-behavior-log-analysis/",
      "type": "study",
      "publisher": "Wislr",
      "verified": "2026-08-20",
      "keyFindings": "Feb 1 – Mar 20 2026 (48 days), 12,099 AI/crawler bot requests (16.9% of 71,603 total), from 288,566 CDN log files, 19 bots identified (GPTBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, OAI-SearchBot, PerplexityBot, Meta-WebIndexer, Bingbot, Googlebot, Applebot, Amazonbot, Bytespider, others). Verbatim: 'No AI bot requests /llms.txt or /llm.txt. Zero hits from GPTBot, ClaudeBot, PerplexityBot, or any other AI crawler.' The only requester was Dataprovider.com (analytics firm): 3 requests."
    },
    {
      "id": "wordlift-homepage",
      "title": "WordLift",
      "url": "https://wordlift.io/",
      "type": "vendor-doc",
      "publisher": "WordLift",
      "verified": "2026-08-20",
      "keyFindings": "Knowledge-graph / schema.org generation platform now marketing an 'AI Visibility Audit' for ChatGPT, Perplexity and AI Overviews, plus agentic 'WordLift Agent' products. Evidence presented is outcome marketing ('80% Traffic Surge, 30% Sales Boost') via case studies; no published methodology, rule list, weights, or per-check sourcing."
    },
    {
      "id": "wpt-webmcp-declarative",
      "title": "web-platform-tests: webmcp/declarative/ conformance suite",
      "url": "https://github.com/web-platform-tests/wpt/tree/master/webmcp/declarative",
      "publisher": "web-platform-tests project",
      "type": "repo",
      "keyFindings": "17 test files authored by dom@chromium.org. getTools-declarative-schema.https.html uses <form toolname=\"search_tool\" tooldescription=\"Search the web\"> with toolparamdescription inputs and asserts the browser-synthesized JSON Schema and tool.name/tool.description. Confirms browser-level implementation, not just a proposal.",
      "verified": "2026-08-20",
      "accessed": "2026-08-21"
    },
    {
      "id": "wtseo-facts-vs-theories",
      "title": "AIO/GEO: Separating Facts from Theories in AI Search",
      "url": "https://www.womenintechseo.com/knowledge/ai-search-optimisation-facts-vs-theories/",
      "type": "article",
      "publisher": "Women in Tech SEO",
      "verified": "2026-08-20",
      "keyFindings": "Useful practitioner-side evidence triage. Classifies clear content structure as evidenced ('lists and structured formatting improved citation recall by up to 40%'; 'bullet-point formats consistently outperformed plain paragraphs when LLMs were asked to make choices'; 'use clear headings and subheadings to make content modular'). Notes FAQ schema is 'still valuable for AIO' despite reduced SEO value 'since its Q&A format maps directly to how LLMs generate answers' — an argument from plausibility, not measurement. Classifies llms.txt as 'still very experimental' and Markdown as speculative: 'No LLM provider has confirmed Markdown is used as a signal'; 'there's no hard evidence that Markdown outperforms clean, semantic HTML.'"
    },
    {
      "id": "x-cards-docs-dead",
      "title": "X Cards documentation URLs (dead-link check)",
      "url": "https://developer.x.com/en/docs/x-for-websites/cards/overview/abouts-cards",
      "type": "vendor-doc",
      "publisher": "X Corp",
      "verified": "2026-08-20",
      "keyFindings": "COUNTER-EVIDENCE, verified 2026-08-20 by HTTP trace. All historical Cards documentation deep links — /cards/overview/abouts-cards, /cards/overview/summary-card-with-large-image, /cards/guides/getting-started, on both developer.twitter.com and developer.x.com — now 301 to the generic https://docs.x.com/overview. docs.x.com/x-api/cards and docs.x.com/fundamentals/x-cards return 404. cards-dev.twitter.com/validator redirects to an X login wall. There is no live, publicly readable X Cards specification in 2026."
    },
    {
      "id": "youbot-docs",
      "title": "YouBot — You.com documentation",
      "url": "https://you.com/docs/youbot",
      "type": "vendor-doc",
      "publisher": "You.com",
      "verified": "2026-08-20",
      "keyFindings": "docs.you.com/youbot 301s here. UA: 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod) Chrome/X.X.X.X Safari/537.36'. 'YouBot fully respects robots.txt directives, including user-agent specific rules and crawl-delay settings' — one of the few vendors explicitly claiming crawl-delay support. Canonical block: 'User-agent: YouBot' / 'Disallow: /'. robots.txt is cached ~30 minutes."
    },
    {
      "id": "zyppy-ai-citation-factors",
      "title": "AI Citation Ranking Factors Analysis",
      "url": "https://signal.zyppy.com/p/ai-citation-ranking-factors",
      "type": "article",
      "publisher": "Cyrus Shepard, Zyppy",
      "verified": "2026-08-20",
      "keyFindings": "Meta-analysis of ~54 published experiments, studies and patents across ChatGPT, Gemini and Perplexity, scored 2.0–9.5 on repeatability, strength of evidence and official documentation support. Relevant scores: 'AI-ready Structure' (headings, sections, tables) 8.6; 'Factually Specific' (concrete numbers over vague statements) 8.3; freshness/dates 7.0; word count 6.7 (inconsistent — longer content performed better but risked incomplete retrieval); schema/structured data 5.6 with effect described as 'typically small'; llms.txt 2.0 (deemed unreliable). FAQ format and meta description were not measured at all — an evidence gap, not a null result."
    }
  ]
}
