Identity

User-Agent

UCPCheckerBot/1.0 (+https://ucpchecker.com/methodology)

Operator

UCP Checker (about) — independent; not affiliated with Google, Shopify, or the UCP consortium

Verification

Web Bot Auth (RFC 9421 HTTP Message Signatures). Our Ed25519 public key is published in a signed key directory at /.well-known/http-message-signatures-directory, and every request UCPCheckerBot sends is signed, so it can be verified cryptographically rather than by IP or User-Agent string.

Documentation

Methodology · Contact

What it fetches

UCPCheckerBot reads what a store publishes for AI agents, plus the few public pages needed to check those declarations. It never logs in, submits forms, adds to a cart, creates a checkout or pays.

• /.well-known/ucp — the UCP profile, the main subject of every check
• /robots.txt (always first), /llms.txt, /sitemap.xml, the homepage, and agent discovery files a store publishes (for example its skills index)
• Documents a profile declares: spec and schema URLs, policy pages, and for stores declaring identity linking, OAuth / OpenID metadata (RFC 8414, RFC 9728) and the declared JWKS
• Read-only calls to the MCP endpoint a profile declares: tools/list, and catalogue, product and policy searches for a small product sample
• A weekly link check: one product page (Shopify) or up to three linked pages and three images (other platforms) per store, each fetched as a one-byte range request
• Favicons, for the store's status page

Daily checks

• Which stores: stores listed on UCP Checker, each with a public status page.
• Low volume: a handful of requests per store per check, roughly once a day per store; the documents above are fetched on their own, slower cycles (weekly or when they change).
• Rate-limited per platform: requests to shared commerce platforms are paced as one stream across all their stores (Shopify-hosted stores, for example, at about 20 store checks a minute in total), and the whole crawl backs off when a platform signals a rate limit.
• Deduplicated storage: responses are content-addressed — unchanged documents are not re-stored.
• Fail-safe backoff: repeatedly failing endpoints are paused for 24 hours.

Monthly market read

Once a month UCPCheckerBot also reads a broad set of public websites, to measure how widely UCP is adopted across online commerce. This is separate from the daily checks, runs from its own server, and takes a few days at the start of each month.

• Three requests per site, once a month: /robots.txt first, then /.well-known/ucp, then the homepage. A site that timed out gets one slower second attempt later the same month.
• Gentle on each site: one request at a time to any one site; under 30 requests a second in total, spread across many thousands of sites; Shopify-hosted stores at no more than 15 requests a minute in total.
• robots.txt first, strictly: if robots.txt disallows UCPCheckerBot, or does not answer, nothing else is requested from the site; a rule that disallows only the homepage is honoured too.
• Aggregate use only: results feed adoption statistics. Sites read only this way get no individual page on UCP Checker.

Signed, every request

Every request UCPCheckerBot makes — daily checks and the market read alike — carries a Web Bot Auth signature (RFC 9421) for the host it is sent to, verifiable against our key directory. A request that claims to be UCPCheckerBot without a valid signature is not ours.

How the results are used

For reference only: conformance status pages, adoption statistics and the reports a store requests about itself. Content is not used to train AI models, and nothing is resold as page content.

Opting out

Add this to your robots.txt and UCPCheckerBot will stop crawling your site (its status on UCP Checker will show as blocked rather than validated):

To allow validation of your UCP profile while blocking everything else:

Questions, removal requests, or anything else: contact us.