llms.txt and robots.txt in Shopify: How to Control Which AI Crawlers Read Your Shop

Whether AI crawlers are allowed to read your Shopify store is solely determined by the robots.txt file. The llms.txt, however, cannot block access: it is a curated table of contents that shows AI systems your most important pages. Our recommendation: Release bots that provide AI answers with live content (e.g., ChatGPT-User, PerplexityBot); otherwise, your store will not appear in AI answers. Whether bots for AI training (e.g., GPTBot, ClaudeBot) get access is your conscious decision.

Before any GEO measure can take effect, a basic prerequisite must be met: AI systems must be able to reach your store at all. This is precisely where the most impactful, often unnoticed, errors occur in practice.

In this guide, we will show you, based on our research and development practice, which AI crawlers you need to know, how to properly set up robots.txt and llms.txt in Shopify, and where hidden blockers lurk. This article deepens step 7 from our guide GEO for Shopify: 9 Steps for Visibility on ChatGPT & Co.

1. robots.txt vs. llms.txt: Who controls what?

The division of roles is clear: robots.txt regulates access, llms.txt provides context. Both files are located in the root directory of your store, but serve fundamentally different purposes.

The robots.txt is the bouncer. It determines which crawlers are allowed to enter your store and which areas are blocked for them. Every serious crawler, whether Googlebot or AI bot, retrieves this file before reading a page and adheres to its rules.

The llms.txt, on the other hand, is a table of contents for AI systems: a curated overview of the most important content on your website in Markdown format. It helps LLMs with limited context windows to find the relevant pages of your store faster and cite them correctly. It cannot allow or prohibit access.

Why this distinction is crucial for your AI visibility

In our research, we have repeatedly encountered the same misunderstanding: llms.txt is presented as a control instrument that can be used to control how AI systems handle content. This is wrong, and the error has consequences in both directions. Anyone who believes that they can keep content out of AI answers via llms.txt has in fact blocked nothing. And conversely, anyone who wants to improve their visibility but blocks the wrong bots in robots.txt will find that even the best llms.txt is useless.

The rule of thumb is: first clarify access via robots.txt, then optimize context with llms.txt. We will follow this order in this guide.

2. These AI crawlers you should know: Retrieval Bots vs. Training Bots

Not all AI crawlers do the same thing. There are two groups of bots with fundamentally different tasks, and they warrant different decisions.

Bot	Operator	Function
OAI-SearchBot	OpenAI	Retrieval (ChatGPT Search)
ChatGPT-User	OpenAI	Retrieval (user queries)
PerplexityBot	Perplexity	Retrieval
Google-Extended	Google	Retrieval / AI Features
GPTBot	OpenAI	Training
ClaudeBot	Anthropic	Training
CCBot	Common Crawl	Training (data collection)

Retrieval and Answer Bots: Blocking means disappearing from AI answers

Retrieval bots fetch content the moment an AI system answers a specific user query. If someone asks ChatGPT for a product recommendation, the OAI-SearchBot retrieves relevant pages live. If these bots are blocked, your store simply cannot appear in AI answers, regardless of the quality of your content. If you want to build AI visibility, these bots must have access.

Training Bots: Exclusion is legitimate, but a conscious decision

Training bots collect content for training future AI models. There is no universal right or wrong here: some brands do not want their content to be included in training data, and that is a legitimate decision. The important thing is that it is made consciously and documented so that it remains traceable during relaunches and audits.

We often observe this error: Those who only want to exclude AI training resort to sweeping rules and accidentally block retrieval bots as well. The result is the opposite of what was desired: the content is not included in training, but the store disappears completely from AI answers. Therefore, always check bot by bot, never with collective rules.

3. Checking and adjusting robots.txt in Shopify

Shopify automatically generates the robots.txt, and the default is GEO-friendly: AI crawlers have access to your store by default. You can find the file at yourshop.com/robots.txt. The first step is always to look at this specific file: Are bots from our table blocked there, and if so, is that intentional?

For adjustments, Shopify provides the template robots.txt.liquid. With this, you can specifically exclude or allow individual bots, for example, if a customer wants to block training bots like GPTBot and ClaudeBot but keep retrieval bots open. Our recommendation: Changes to robots.txt should be documented and re-checked after every major relaunch, as individual rules are often lost in the process.

Caution with third-party scripts

A discovery from our research that surprised us: In several shops, the Googlebot was blocked by robots.txt without anyone deliberately setting this rule. The cause was apparently the code of a third-party app that had affected the file.

The consequence of this: Don't rely on your robots.txt looking the way you left it. Apps and scripts can introduce rules you never intended. Therefore, check the file regularly, especially after installing new apps, and include it in every technical audit. The Google Rich Results Test also detects such blockages, as we show in our guide to structured data in Shopify.

4. Setting up llms.txt in Shopify

Since the end of May, Shopify has offered its own templates for llms.txt: llms.txt.liquid and llms-full.txt.liquid. This allows the file to be managed directly in the theme and dynamically populated with store data, without external apps or manual uploads.

The basic principle of a good llms.txt: curated rather than complete. The file is a Markdown overview that specifically provides LLMs with the most important content of your website. The most common mistake is to list every product entry and every blog post. LLMs work with a limited context, and an overloaded file buries precisely the pages that should be cited. It should contain:

• Shop description: What the brand sells, for whom, what distinguishes it
• Most important collections: The central categories instead of the entire catalog
• Central content pages: Guides, pillar pages, FAQ pages
• Relevant policies: Shipping, returns, warranty

Realistic expectations: The standard is young

As clear as our recommendation for setup is, the assessment must be equally honest: llms.txt is a young standard and is not yet reliably observed by all LLM providers. The file improves the capture and citability of your content where it is read, but it is not a guarantee for AI visibility and does not replace any of the technical fundamentals such as rendering, structured data, or crawler approvals.

Our assessment: With Shopify templates, the effort is now so low that setup is still worthwhile. Anyone who sets up the file correctly now will be prepared when the standard becomes more widely adopted and risks nothing by not doing so.

5. Agentic Storefronts: Why the backend offers no crawler control

Shopify has created its own area in the admin for Agentic Storefronts or AI Sales Channels, through which shops are intended to become visible in AI channels such as ChatGPT. It would be obvious to assume: whoever deactivates the channel also excludes the AI crawlers. This is precisely not the case.

The insight from our research: Activating merely creates a structured storefront entry in the backend; deactivating only removes this entry. The shop remains fully visible and readable for AI crawlers. Anyone who actually wants to block access cannot avoid a code adjustment in robots.txt. The backend here creates the illusion of control it does not possess.

In practice, this means: Treat Agentic Storefront settings and crawler control as two separate issues. The former regulates participation in Shopify's AI channels, the latter access to your content, and only robots.txt determines access.

Regarding the feature itself: According to our tests, it is currently regionally limited, in parts intransparent, and not yet mature for the German market. We continuously monitor its development but do not currently recommend it as an active optimization lever.

6. When robots.txt allows, but still nothing arrives: WAF and Bot Protection

There's a blocker that robots.txt can't even see: security layers like Web Application Firewalls (WAF), bot protection, or CDN rules. They sit in front of your shop and can reject AI crawlers, even though robots.txt explicitly allows access. The crawler then receives an error code instead of the page, such as 403 (access denied) or 429 (too many requests).

The insidious thing about it: From the outside, everything looks correct. The robots.txt is clean, permissions are correct, and yet the content never reaches the AI systems. Bot protection rules, in particular, are often trained to throttle automated traffic across the board, and AI crawlers are exactly that from their perspective.

Therefore, every thorough check includes a live access test: For each relevant bot, check which status code your shop actually returns. Only a 200 status code confirms that the crawler is truly getting through. In our GEO checklist, this test is anchored as a separate checklist item, documented as a status code protocol for each bot.

Our recommendation: For every setup, clarify whether a WAF, bot protection, or CDN is in front of the shop and with what rules. With pure Shopify standard setups, this is rarely a problem, but as soon as additional security services are involved, the live test is mandatory.

7. Common errors in AI crawler control

Most visibility issues we encounter with this topic stem from five recurring errors:

1. Outdated bot names: The robots.txt refers to crawler names that no longer exist, such as "anthropic-ai" instead of ClaudeBot. The rule then goes unheeded, and the actual bot remains uncontrolled. Check the current bot names directly in the providers' documentation.
2. Blanket blocking rules: Collective rules that treat all AI bots the same affect retrieval and training bots equally. Anyone who only wanted to exclude training will thus disappear completely from AI answers. Always decide bot by bot.
3. llms.txt misunderstood as a blocker: The llms.txt cannot prevent access. Anyone who wants to keep content out of AI systems and relies on llms.txt for this has in fact blocked nothing.
4. Backend switches as pseudo-control: Deactivating Agentic Storefronts in the Shopify admin does not block AI crawlers. The only effective access control is robots.txt.
5. Unchecked security layers: WAF, bot protection, or CDN rules block crawlers despite a clean robots.txt. Without a live access test with status code verification, this blockage remains invisible.

Our recommendation: Include these five points as a fixed checklist block and repeat the check after relaunches, app installations, and changes to the security setup. Crawler control is not a one-time setup, but a state that can change unnoticed.

8. Conclusion: Make conscious decisions instead of accidental blocking

Controlling AI crawlers is not complex technology, but it requires conscious decisions. Most shops today block or allow randomly, due to legacy issues, blanket rules, or app code. This is how you should proceed instead:

1. Check: Open yourshop.com/robots.txt and compare the entries with the current bot names. Also clarify whether WAF, bot protection, or CDN are in front of the shop.
2. Decide: Allow retrieval bots if you want AI visibility. Make a conscious decision for training bots, bot by bot, instead of blanket rules.
3. Document: Record every decision and every change to robots.txt so that it survives relaunches and team changes.
4. Test: Verify via live access test that authorized bots actually receive status code 200. Repeat the check after relaunches and new apps.
5. Provide context: Then set up llms.txt using the Shopify templates, curated rather than complete.

This settles access, the basic prerequisite for everything else. Whether AI systems then correctly understand and cite your content depends on the next levels: clean rendering, structured data, and answer-first content. We show how all measures interact in our guide GEO for Shopify: 9 Steps for Visibility on ChatGPT & Co.