The 2026 Robots.txt Playbook for AI Crawlers
Your robots.txt file is now one of the most consequential lines of "code" in your entire marketing stack. Get it wrong and you can quietly delete yourself from ChatGPT, Perplexity, or Google's AI features overnight. The catch in 2026 is that "AI crawler" isn't one thing - some bots train models, others fetch pages to cite you live, and you usually want to treat those two groups very differently.
Quick answer: The 2026 best practice is to allow the search and user-fetch bots that cite content (OAI-SearchBot, PerplexityBot, ClaudeBot, Googlebot) while choosing whether to block training-only bots (GPTBot, Google-Extended, Applebot-Extended). Blocking everything makes you invisible in AI answers; blocking nothing feeds model training. Decide per-bot, then verify in your server logs.
What is the difference between training bots and search bots?
The key distinction is purpose. Training bots gather content to train future models - blocking them keeps your work out of the training set but doesn't affect whether you're cited today. Search and user-fetch bots retrieve pages to answer a live query and can name you in the response. If your goal is AI visibility, search bots are the ones you must let in.
Which AI crawlers should you know in 2026?
A handful of user agents account for most AI crawling. The table below groups the major ones by purpose so you can decide which to allow. Note that some companies run separate bots for training and search - blocking one doesn't block the other, which is exactly what makes selective control possible.
| Crawler | Run by | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training data collection |
| OAI-SearchBot | OpenAI | Live search / citations |
| ClaudeBot | Anthropic | Crawl + web search (volume up ~800% early 2026) |
| PerplexityBot | Perplexity | Live search / citations |
| Google-Extended | AI training / AI features opt-out control | |
| Applebot-Extended | Apple | AI training control |
Key Insight
Blocking GPTBot keeps your content out of OpenAI's training data - but it's OAI-SearchBot that fetches pages for live ChatGPT answers. Confusing the two is the most common way brands accidentally stay visible in training they didn't want, or disappear from search they did.
The block + allow template
Here's a sensible default for a brand that wants AI citations but prefers to opt out of pure model training. Adjust to taste - if you're happy to contribute to training, simply remove the Disallow lines. Always keep Googlebot fully allowed so your classic search rankings are untouched.
Does blocking AI crawlers hurt my Google SEO?
Blocking AI-specific crawlers like GPTBot or Google-Extended does not affect your classic Google Search ranking, which depends entirely on Googlebot. The one caveat: blocking Google-Extended removes you from Google's AI-powered features. So never block Googlebot, and only block Google-Extended if you knowingly want to sit out Google's AI answers.
How to verify it's working
Don't trust the file blindly - confirm reality in your logs. Filter server access logs by user agent to see which bots are actually fetching pages and how often. This is also how you'd notice, say, ClaudeBot's reported ~800% volume jump in early 2026, or catch a misconfiguration that's quietly blocking a bot you meant to allow.
- 1 Decide your stance: Cited everywhere, or cited-but-no-training? That choice drives every line.
- 2 Allow the citation bots: OAI-SearchBot, PerplexityBot, ClaudeBot, and Googlebot stay open.
- 3 Block training bots only if you mean it: GPTBot, Google-Extended, Applebot-Extended.
- 4 Verify in logs: Confirm the right bots are crawling and the blocked ones stopped.
- 5 Re-check quarterly: New bots launch constantly; an old robots.txt silently misses them.
"If you've blocked GPTBot in your robots.txt, your content won't appear in ChatGPT's training data or web search results."
-- AI crawler management guidance, 2026
Frequently asked questions
Should I block AI crawlers in robots.txt?
It depends on your goal. If you want AI visibility, you must allow the search and user-fetch bots that cite content, such as OAI-SearchBot and PerplexityBot. You can separately block training-only bots like GPTBot or Google-Extended if you don't want your content used for training. Blocking everything removes you from AI answers.
What is the difference between training bots and search bots?
Training bots collect content to train future models (for example GPTBot and Google-Extended), while search or user-fetch bots retrieve pages to answer a live query and can cite you (for example OAI-SearchBot and PerplexityBot). Allowing search bots while optionally blocking training bots is the common 2026 strategy.
Does blocking AI crawlers hurt my Google SEO?
Blocking AI-specific crawlers like GPTBot or Google-Extended does not affect classic Google Search ranking, which depends on Googlebot. However, blocking Google-Extended can remove you from Google's AI features. Never block Googlebot itself if you want to keep ranking in regular search.
How do I know which AI crawlers are visiting my site?
Filter your server access logs by user agent. Look for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, and Applebot-Extended. ClaudeBot volume reportedly rose around 800% in early 2026, so checking logs tells you who is really crawling and how often.
Figures in this article come from third-party industry research published in 2025-2026. We summarize and link the originals below; numbers are directional findings from those studies, not guarantees.