What is robots.txt?

The robots.txt file tells web crawlers which parts of your store they can and can't access. For AI commerce, it controls whether AI crawlers like GPTBot, ClaudeBot, and Google-Extended can discover your products.

Why it matters

• Blocked crawlers mean invisible products — if AI crawlers are disallowed, your store won't appear in AI shopping assistants, comparison engines, or voice results
• Many stores block AI crawlers by accident — a misconfigured robots.txt or a generic "block all" rule can silently exclude your entire catalog
• Explicit allowlisting is safer than defaults — AI crawlers are newer than most robots.txt files, so they often aren't covered by existing rules

What Lantern does

Lantern ensures your robots.txt explicitly allows known AI crawlers access to your product catalog while keeping admin and checkout pages protected.

Related

You should also set up an llms.txt file to provide AI agents with a curated map of your store.