Your robots.txt file was probably set up years ago to manage Googlebot and a few other crawlers. But the set of bots that matter for your business has grown significantly. AI shopping assistants — GPTBot, ClaudeBot, PerplexityBot, and others — all send their own crawlers to learn about products. If your robots.txt is blocking them, your products are invisible to those assistants, and you would have no idea.
How Robots.txt Works (and Why AI Bots Obey It)
Robots.txt is a plain-text file at yourdomain.com/robots.txt that tells crawlers which parts of your site they are and are not allowed to access. It is not enforced technically — it is a convention that well-behaved bots follow voluntarily.
The major AI companies have publicly committed to respecting robots.txt instructions. OpenAI's GPTBot, Anthropic's ClaudeBot, Perplexity's PerplexityBot, Google's Google-Extended, and Amazon's Amazonbot all check your robots.txt before crawling. If your file blocks them — either intentionally or by a catch-all rule — they will not index your products and cannot use them in recommendations.
The most common way ecommerce stores accidentally block AI crawlers is with a Disallow: / rule under User-agent: *. This was sometimes added to staging sites or during development and never removed. It blocks every bot that is not explicitly allowed — including all the AI shopping crawlers.
The AI Crawlers You Should Be Allowing
Each major AI assistant uses its own named bot. Here are the ones that matter most for ecommerce product visibility:
- GPTBot — OpenAI's crawler, used to train and update ChatGPT's shopping knowledge
- OAI-SearchBot — OpenAI's real-time search crawler, used for ChatGPT's live web browsing
- ClaudeBot — Anthropic's crawler for Claude's knowledge base
- PerplexityBot — Perplexity's crawler, used to power its shopping recommendations
- Google-Extended — Google's opt-in/opt-out bot for Gemini and AI Overviews training
- AmazonBot — Amazon's crawler, relevant if you sell on or alongside Amazon
- CCBot — Common Crawl's bot, used as a data source by several AI providers
The safe default for most ecommerce stores is to allow all of these. Unless you have a specific legal or competitive reason to block AI crawlers, allowing them costs you nothing and ensures your products remain visible in AI-driven shopping surfaces.
Our free audit checks your robots.txt against all major AI crawlers and flags any that are being blocked.
Run Free Audit →What a Correct Robots.txt Looks Like
A well-configured robots.txt for an ecommerce store allows all major AI crawlers, blocks genuinely problematic paths like checkout and account pages, and points to your sitemap. Here is a minimal correct example:
- Allow all crawlers on product, collection, and content pages
- Disallow checkout, cart, account, and admin paths (these have no SEO or AI value and contain sensitive session data)
- Include a
Sitemap:directive pointing to your sitemap.xml - If you use a catch-all
User-agent: *rule, make sure it is not blocking more than necessary
On Shopify, the platform generates a default robots.txt that correctly handles most of this. However, some third-party apps and custom themes modify it — sometimes incorrectly. It is worth checking your live robots.txt file directly, even on Shopify.
The Google-Extended Decision
Google-Extended is a separate bot from Googlebot. It is used specifically for training Google's Gemini models and powering AI Overviews. Some publishers have chosen to block it as a way of controlling how their content is used for AI training.
For ecommerce stores, this decision is almost always the wrong one. Blocking Google-Extended does not prevent your products from appearing in regular search results — Googlebot still crawls normally. But it does reduce the likelihood that your products appear in Google's AI-powered shopping recommendations and AI Overviews, which are increasingly the first thing shoppers see on high-intent queries. For the vast majority of stores, the right call is to allow Google-Extended.
Frequently Asked Questions
If I block AI crawlers in robots.txt, will I disappear from ChatGPT completely?
Not immediately. AI models are trained on snapshots of the web taken at intervals. If you have been crawled in the past and your products appear in training data, that knowledge persists until the model is retrained. However, for real-time search features like Perplexity or ChatGPT's web browsing, blocking the crawler means your current pages will not be accessible.
Can AI crawlers ignore my robots.txt?
Major AI companies have publicly stated they respect robots.txt. OpenAI, Anthropic, Google, and Perplexity have all confirmed their bots follow these conventions. Smaller or less reputable bots may not comply, but those are not the ones that power mainstream AI shopping assistants.
Does allowing AI crawlers slow down my site?
No. Crawler traffic is separate from user traffic and does not affect your page load speed or Core Web Vitals scores. The crawl rate of AI bots is also generally low and considerate — they are not hammering your server the way a poorly configured scraper might.
Check Your AI Crawler Access in 30 Seconds
Run a free audit to see exactly which AI bots your store is allowing and which it is blocking — along with your full SEO and AI readiness scores.
Run Free AuditNo credit card · Takes ~30 seconds