robots.txt for AI Bots

Blocking every AI crawler in your robots.txt is no longer the safe default for most commercial sites. The key distinction is between training crawlers and retrieval crawlers: blocking retrieval bots removes your site from AI answers and citations, while blocking training bots only affects whether your content feeds future models. Allow retrieval bots almost universally, and decide on training bots based on your business model.

Why "block all AI" backfired

The 2024 instinct to block every AI bot to protect content has reversed for most commercial sites. That advice was overcautious, and it cost sites measurable AI-Mode and ChatGPT visibility through 2025.

The reason is simple: blocking a training crawler is not the same as blocking a retrieval crawler. Training crawlers gather data that may feed future model versions. Retrieval crawlers fetch your pages live so an AI system can cite them in an answer right now. Block the retrieval crawlers and you disappear from AI citations — a direct hit to visibility.

The lawsuits-versus-licensing landscape has also stabilized since then. That means you can afford to block specifically rather than broadly, keeping the crawlers that drive distribution while restricting only the ones that conflict with your model.

The current AI bot taxonomy

Most major AI bots respect robots.txt. The table below summarizes the operators, purposes, and a 2026 recommendation for each. Note that some naming has shifted over time — verify current user-agent strings before you commit rules.

A couple of caveats worth remembering. PerplexityBot mostly respects robots.txt but was caught spoofing in 2024, and Perplexity-User behavior has been inconsistent. Bytespider has historically respected robots.txt poorly.

Decide by business model

The core rule is to allow retrieval bots and decide on training bots based on your business model. Here is how that plays out across common site types.

• Publisher, news, or paywalled content: Block training (GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Meta-ExternalAgent) and allow retrieval (OAI-SearchBot, ChatGPT-User, PerplexityBot, Googlebot). Negotiate licensing deals directly — the NYT/OpenAI, Reddit/Google, and FT/Anthropic arrangements are template examples.
• SaaS, B2B, and docs sites: Allow nearly everything. You want to be the canonical answer when developers ask Claude or ChatGPT how to do something with your product, and training inclusion becomes future free distribution.
• E-commerce: Allow retrieval, which is essential for product discovery in agents. Training is lower-stakes because your inventory changes faster than training cycles anyway.
• Affiliate or content sites: Mixed. Allow retrieval, but block training if your moat is the writing itself.

If you run a documentation-heavy or product-led site, being cited well ties directly into AEO & GEO and the shift toward The Zero-Click, Agent Future.

A sample SaaS robots.txt

For a SaaS site that wants maximum AI exposure, a workable pattern looks like this. Notice that retrieval bots are named explicitly, because some bots only crawl when directly addressed.

User-agent: *
Allow: /

# Block low-value training crawlers
User-agent: Bytespider
Disallow: /

# Explicit allows for retrieval (some bots only crawl if explicitly named)
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://example.com/sitemap.xml

This is a starting point, not a universal answer — adjust the training-bot rules to match whichever business model above fits your site.

Enforcement is a separate problem

robots.txt is honor-system guidance. Compliant bots read it and behave, but it does not physically stop anything. If you need a hard block — for example, on a training crawler that ignores robots.txt or spoofs its user-agent — you enforce at the WAF level.

Cloudflare's AI Audit and bot-blocking features (launched in 2024) and similar WAF-level controls are how you actually prevent access. Use robots.txt to state your preference to well-behaved bots, and use those controls when you need enforcement that does not depend on cooperation.

For broader context on how crawling and indexing fit together, see Technical SEO. If you are also weighing whether a separate AI-guidance file helps, read llms.txt: The Reality Check.

What to do

1. Classify each AI bot as training or retrieval before writing any rules, using the taxonomy above.
2. Allow retrieval bots — OAI-SearchBot, ChatGPT-User, PerplexityBot, Googlebot, and the current Claude retrieval UA — so you stay eligible for AI citations.
3. Match your training-bot rules to your business model: allow broadly for SaaS/B2B and e-commerce; block training for publishers and moat-driven content sites.
4. Name retrieval bots explicitly in robots.txt, since some crawl only when directly addressed.
5. Block Bytespider and other poorly-behaved crawlers unless you specifically want that exposure.
6. Verify current user-agent strings before deploying, because naming shifts over time (especially for Anthropic's bots).
7. Add WAF-level controls for any bot you need to hard-block, since robots.txt alone cannot enforce.
8. Revisit publisher-style blocks against licensing options, using existing deal templates as a reference point.