A robots.txt file sits at the root of every website and tells automated visitors which parts of the site they are allowed to read. On two of the four Shopify stores we checked it now does something else as well: it hands AI shopping agents the address of a machine-readable version of the catalogue, the cart and the checkout. Most owners have never opened the file, and nobody asked them to switch this on.
Key takeaways
- Two of the four Shopify stores we checked serve a robots.txt that opens with instructions addressed to AI agents, pointing at an
agents.md, a discovery document at/.well-known/ucpand a live commerce endpoint at/api/ucp/mcp. See it yourself on granadillaswim.com/robots.txt. - That endpoint declares real capabilities: catalogue search, catalogue lookup, cart, discounts, order, fulfilment and checkout, with named payment handlers (the discovery document is public).
- The same file draws a hard line at payment. In its own words: "Checkouts are for humans. DO NOT complete checkout, payment, or order placement automatically."
- The two stores that did not have it are both large brands whose robots.txt has clearly been customised. Editing that file can remove the agent interface without anyone noticing.
- Across 15 readable South African retail sites, nine have no rules for AI crawlers at all, three name them and allow them, one blocks seven of them outright, and two take a mixed approach.
What is actually in the file
Open granadillaswim.com/robots.txt and the first thing you meet is not a crawler rule. It is a note written for software:
"Shopify storefront. Public product, collection, page, blog, policy, cart, and localized HTML is crawlable. Agent instructions: /agents.md. UCP discovery: /.well-known/ucp. UCP/MCP endpoint: /api/ucp/mcp. Agents should use UCP/MCP for catalog, cart, and checkout. Payment requires buyer approval."
The contact address at the bottom of that block is [email protected], which tells you whose file it is. We found the same preamble, near enough word for word, on sealandgear.com/robots.txt. Neither merchant wrote it.
What the store is offering to do
Follow the discovery link and the store publishes a profile of what an agent may negotiate with it. Granadilla Swim's declares eight capabilities — catalogue search, catalogue lookup, cart, discount, order, fulfilment, checkout, and a Shopify catalogue extension — along with two payment handlers and the protocol versions it speaks. Shopify's own developer documentation on agent profiles describes the mechanism: an agent declares what it can do, the shop declares what it supports, and the two intersect to a negotiated set for the session.
The agents.md file fills in the practical detail: which URLs return product JSON without authentication, how to search, how to back off when rate-limited, and to pass country and currency so the prices quoted are right. It also states the rule that matters most, that agents must not complete payment without explicit buyer consent.
The plumbing being built is for discovery and cart assembly, with a human approving the payment. It matches what OpenAI told merchants when it stepped back from selling inside the chat window, which we covered in the second audience.
What South African stores are actually telling crawlers
We read the robots.txt of seventeen South African retail sites on 27 August 2026. Two would not return a readable file. Of the fifteen that did, nine have no rules for AI crawlers whatsoever, so those crawlers fall under the general rule and are free to read the site. Three name them and let them in: Woolworths lists a dozen by name and allows each one, while blocking Sogou, Baiduspider, Yandex and several others. Checkers does much the same. OneDayOnly names six and leaves them unrestricted.
One site goes the other way. Incredible Connection blocks GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, meta-externalagent and Amazonbot with a full disallow, served through what appears to be a managed rule at the network layer rather than a hand-written file. Faithful to Nature blocks Bytespider and Meta's agents and leaves the assistants alone. Superbalist names ClaudeBot and then gives it exactly the same rules as everyone else.
This is a spot check on fifteen sites, not a survey, and it should be read as one. What it does show is that the fear of having been quietly shut out of AI answers is mostly unfounded here. If a South African store is missing from an AI recommendation, the likely reason is that the answer engine could not find enough accurate information to work with, which points at the information on the site rather than the permissions around it.
One distinction that changes the decision
Not every bot with AI in its description does the same job. GPTBot gathers material for training. OAI-SearchBot and ChatGPT-User fetch pages to answer a question someone is asking right now. Blocking the first is a defensible choice about your content. Blocking the second removes you from the answer while the customer is standing there with their wallet out. They are separate lines in the file and they deserve separate decisions. A single toggle labelled "block AI bots" does not offer you that choice.
Frequently asked questions
What is robots.txt?
A plain-text file at the root of a website, reachable by adding /robots.txt to the domain, that tells automated visitors which parts of the site they may read. It works by naming a user-agent and then allowing or disallowing paths for it. It is a set of instructions rather than a lock, and well-behaved crawlers follow it.
Does my Shopify store have an AI agent interface?
Possibly, and you can check in under a minute. Open your domain followed by /robots.txt. If the file begins with a comment block naming agents.md, UCP discovery and an MCP endpoint, the interface is live. Two of the four Shopify stores we checked had it; the two that did not were large brands with a customised robots.txt, which suggests editing that file can remove it.
Should I block AI crawlers from my website?
Treat training crawlers and retrieval crawlers as separate decisions. GPTBot collects material to train models, and blocking it is a reasonable choice about your content. OAI-SearchBot and ChatGPT-User fetch pages to answer a question a customer is asking at that moment, and blocking those removes you from the answer. A single toggle that blocks everything with AI in the name does not make that distinction.
Can an AI agent buy something from my store without me knowing?
Not under the rules Shopify publishes. The robots.txt preamble states that checkouts are for humans and that agents must not complete checkout, payment or order placement automatically, and the agents.md file repeats that payment requires explicit buyer consent. The interface is built for discovery, search and cart assembly, with a person approving the payment.
The check worth running today
Open your own domain with /robots.txt on the end and read it. Then try /agents.md and /.well-known/ucp. Two minutes tells you whether your store is legible to an agent or quietly blocked by a setting nobody remembers turning on. If what comes back is thin or wrong, that is the same catalogue and content work that makes a store sell to people: it belongs to an E-Commerce Website Build if the store itself is the problem, and to SEO if the information around it is.