Can search engines — and AI — actually crawl your store?
One misconfigured line in robots.txt can hide your entire store from Google. We check it, your sitemap, and your llms.txt in seconds.
Enter your email to unlock every issue we found, with a specific explanation and fix for each.
We’ll also send occasional Shopify tips and Bespop product updates. Unsubscribe anytime. See our privacy policy.
Five crawlability fundamentals
We fetch your robots.txt, sitemap, and llms.txt exactly the way Googlebot and AI crawlers do, and parse what we find.
robots.txt exists and is reachable
A missing or erroring robots.txt makes crawler behavior unpredictable. Shopify generates one automatically — but customizations and misconfigured apps can break it.
The site-wide block
We parse crawler rules group by group, looking for the killer line: "Disallow: /" applied to all crawlers. It’s rare — usually a leftover from a launch or a botched edit — but it’s catastrophic when present.
Sitemap referenced, reachable & listed
Your robots.txt should point crawlers at your sitemap, the sitemap URL should respond, and it should actually list your pages — we check all three and show you every URL we find.
llms.txt for AI crawlers
An emerging convention — like robots.txt, but for AI. It gives ChatGPT, Claude, Perplexity, and other assistants a curated summary of your store. Not required yet, but early adopters get better visibility as AI-powered shopping search grows.
If crawlers can’t read your store, nothing else about SEO matters
Crawlability sits underneath every other SEO effort. You can have perfect product pages, great reviews, and strong backlinks — and none of it counts if robots.txt tells Google to stay out. These failures are rare but uniquely devastating because they’re silent: your store works perfectly for humans while organic traffic quietly decays. The classic incident is a "Disallow: /" added during development that ships to production; stores have lost months of rankings this way before anyone thought to look.
Your sitemap is the flip side: it’s how Google discovers new products quickly and notices updates. A broken sitemap means new arrivals can take weeks to appear in search instead of days — a real cost when you’re launching seasonal collections.
What healthy crawlability delivers
Insurance against silent catastrophe
A crawl block is the one SEO problem that can erase all your other work. Thirty seconds of checking rules it out.
Faster indexing of new products
A clean sitemap gets new arrivals into search results in days, which matters most for seasonal drops and trend-driven catalogs.
Efficient crawl budget
Clear rules and an accurate sitemap keep Google’s crawler focused on pages that matter instead of wandering into dead ends.
A baseline for AI search too
AI assistants and answer engines respect the same crawl rules. A store that’s open to crawlers is discoverable everywhere.
Three steps, about ten seconds
Paste your store URL
Any public storefront works — yourstore.com or the myshopify.com address. No login, no install, no admin access.
We scan what visitors see
Our scanner reads your public storefront the same way a browser and a search engine do, and runs every check server-side.
Get your score & full report
Your score and top issue appear instantly. Unlock the full issue-by-issue report — with a specific fix for each finding — with your email.
Common questions
Shopify manages robots.txt automatically — can mine really be broken?
Shopify’s default is safe, but it’s been customizable via robots.txt.liquid since 2021 — and edits, tutorials followed half-way, and certain apps can introduce harmful rules. The default is fine; the customizations are what we’re checking.
The scan says my store blocks specific bots. Is that a problem?
Usually intentional and fine — many stores block scrapers or aggressive SEO crawlers. We only raise the critical flag when the wildcard group (all crawlers, including Google) is blocked from everything.
My sitemap "could not be verified" — what does that mean?
Your store’s bot protection throttled our scanner before we could fetch it. It’s not evidence anything is wrong — wait a minute and rescan, and it usually resolves.
Should I submit my sitemap to Google manually?
Worth doing once: add your store in Google Search Console and submit the sitemap URL. You’ll then get Google’s own coverage reports — which pages are indexed, which have problems — for free, forever.
What is llms.txt and do I actually need one?
It’s a plain-text file at yourstore.com/llms.txt that gives AI assistants a curated summary of your site — similar in spirit to robots.txt, but aimed at language models instead of search crawlers. It’s not an official standard yet and most stores don’t have one, so we flag it as informational, not critical. Worth adding if you want an edge in AI-powered shopping search as that channel grows.
Want automatic sitemap submission, an llms.txt for AI search engines, and a safe robots.txt editor with rollback? That’s in Bespop SEO.