Direct answer: There are two fundamentally different categories of AI bot hitting your server right now, training crawlers that scrape content to build future models (GPTBot, Google-Extended), and retrieval bots that fetch content in real time to answer a specific user's question (OAI-SearchBot, ClaudeBot, PerplexityBot). Blocking the wrong one can quietly remove your site from AI-generated answers entirely. Most robots.txt files I audit get this distinction wrong, usually by accident.
I check this on almost every technical audit now because I keep finding the same mistake: a security plugin or a copy-pasted robots.txt rule from years ago blanket-blocking "bots" without anyone realizing it's also blocking the exact crawlers that would let the site get cited by ChatGPT or Perplexity. So let's actually go through this properly.
The Distinction That Matters More Than Any Individual Bot Name
Before the specific bots, understand this split, because it's the actual decision framework, not the individual names.
Training crawlers scrape your content to help train a future version of a foundation model. They don't provide citations, links, or direct traffic. GPTBot and Google-Extended fall into this category.
Retrieval bots fetch your content in real time, often triggered by an actual user's question, to generate an answer right now, and typically do provide a citation or link back. OAI-SearchBot, ChatGPT-User, Claude-SearchBot, and PerplexityBot fall here.
This distinction is the entire decision, not a technical footnote. If your priority is AI-search visibility, blocking training crawlers while allowing retrieval bots is a completely reasonable, common posture. Blocking retrieval bots is the mistake, because that's the one that actually removes you from AI-generated answers.
The Bots, One at a Time
GPTBot is OpenAI's training crawler. It scrapes content to help train future GPT models. It doesn't power ChatGPT's real-time search results, that's a separate bot entirely, which is exactly the confusion that causes most misconfigurations.
Quick Knowledge Check
Which HTTP status code should be used for a permanent URL redirect?
OAI-SearchBot is the bot that actually indexes content for ChatGPT Search results. OpenAI split this out from GPTBot in late 2024 specifically because the two serve completely different purposes. OpenAI's own documentation states plainly that sites blocking OAI-SearchBot will not appear in ChatGPT search answers, though navigational links may occasionally still appear (source). One additional wrinkle worth knowing: OpenAI states that ChatGPT-User, the bot triggered by a live user action inside ChatGPT, may not be governed by robots.txt in quite the same way automated crawlers are.
ClaudeBot and anthropic-ai are Anthropic's crawlers, handling both training and retrieval-adjacent functions depending on configuration. If your goal is Claude citation eligibility, these need to be allowed.
PerplexityBot is Perplexity's primary crawler, with Perplexity-User as a separate real-time agent. Worth knowing honestly: Perplexity's crawlers have been documented ignoring robots.txt entirely in some cases, meaning for non-compliant bot behavior, robots.txt isn't actually a technical barrier at all, just a request. Real enforcement in those cases happens at the server or firewall level, not the robots.txt file.
Google-Extended is Google's specific opt-out mechanism for Gemini and AI Overview training, separate from standard Googlebot. Disallowing Google-Extended doesn't affect your regular Google search ranking, it specifically opts you out of your content being used for Google's AI model training.
CCBot (Common Crawl) and Bytespider (ByteDance/TikTok) round out the list most audits should check. Bytespider in particular has been documented ignoring robots.txt directives as well.
An Important Honesty Check: robots.txt Isn't Actually Law
I think this gets glossed over in most explainers, so let me say it plainly. robots.txt, standardized as RFC 9309, is a voluntary request that well-behaved crawlers choose to honor. It carries no legal enforcement mechanism by itself (source). Googlebot, GPTBot, ClaudeBot, and PerplexityBot are generally documented as compliant. Some other bots, and Perplexity's own crawlers in certain configurations, have been documented not fully complying. For genuinely non-compliant traffic, real enforcement requires server-level or CDN/WAF-level blocking, robots.txt alone won't stop a bot that's decided not to respect it.
Does Blocking AI Crawlers Actually Protect You?
This is the question I get asked most, usually by a client worried about content scraping. The honest answer, backed by real research, is more nuanced than either "always block" or "always allow."
Research from Rutgers and Wharton, published December 2025, found that publishers blocking AI crawlers experienced a 23.1% total traffic decline, without reliably reducing citation rates in the first place (source). In plain terms: blocking often costs real, measurable traffic while not actually achieving the protection it was meant to provide. That's exactly why a selective posture, block training crawlers if you have specific data concerns, allow retrieval bots because that's your actual visibility channel, is what most serious practitioners recommend now.
How has AI Overview affected your or clients' organic traffic quality?
Click to vote • Results shown after voting
This isn't theoretical. Cloudflare Radar's own data, pulled from a snapshot of 4,223 sites in August 2026, shows the real-world split clearly. GPTBot appears in DISALLOW rules on 633 domains and in ALLOW rules on 282, the most contested bot by far. PerplexityBot shows a notably different pattern: 311 domains block it, but 275 allow it, a much higher allow-to-block ratio than GPTBot or ClaudeBot, suggesting site owners are generally more receptive to Perplexity specifically than to OpenAI's or Anthropic's crawlers (source). Technology and business-focused domains publish the most AI-crawler-specific rules of any category, meaning if you're in a technical or B2B space, your competitors are very likely already making deliberate decisions here, not leaving the default configuration untouched.
A Working Example Configuration
Here's the selective posture I typically recommend and implement, blocking training-only crawlers while keeping every meaningful AI-search retrieval path open:
```
Block training-only crawlers
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
Allow AI search retrieval bots
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
Standard search engines
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
```
Adjust based on your own actual data-sharing preferences, there's no single universally correct configuration, but this reflects the posture most consultants recommend when AI-search visibility is a real business priority. I'd also recommend pairing this with an llms.txt file, a newer, complementary standard specifically written for AI systems rather than crawl instructions alone.
How I Check This on Every Audit
I don't just read the robots.txt file and trust it. I check server logs directly to see what these bots are actually doing on the site in practice, verified through reverse DNS lookup rather than trusting the user-agent string alone, since user-agent headers can be spoofed by anyone. This is part of every technical audit I run, and it's exactly the kind of verification step covered in Phase 3 of the Ghost Partner Delivery System.
📌Technical SEO Principles
Frequently Asked Questions
What's the difference between GPTBot and OAI-SearchBot?Direct answer: GPTBot scrapes content to train OpenAI's foundation models and has no connection to real-time search. OAI-SearchBot specifically indexes content for ChatGPT Search results. Blocking GPTBot stops training data collection; blocking OAI-SearchBot removes you from ChatGPT's live search answers.
Is it safe to block all AI crawlers if I don't want my content used for training?Direct answer: It reduces training-data use, but research shows it can cause meaningful traffic decline, around 23.1% in one study, without reliably reducing citation elsewhere. A selective approach, blocking training bots while allowing retrieval bots, is generally the better-tested posture.
Do I need to trust the user-agent name, or can it be faked?Direct answer: User-agent strings can absolutely be spoofed by anyone. Verified confirmation requires a reverse DNS lookup against the requesting IP, not just reading the header string, which is standard practice in any real technical audit.
Does robots.txt legally prevent a bot from crawling my site?Direct answer: No. robots.txt (RFC 9309) is a voluntary standard most major crawlers choose to honor, but it carries no legal enforcement by itself. Some crawlers, including some of Perplexity's, have been documented not fully complying, meaning real protection against non-compliant traffic requires server or firewall-level blocking.
How often does this bot list actually change?Direct answer: Fairly often. New bots get introduced, existing ones get split into more specific variants (the way OpenAI split GPTBot from OAI-SearchBot), and it's worth checking your configuration every few months rather than setting it once and assuming it stays correct.
Getting this configuration wrong is one of the most common, and most invisible, ways a technically solid site quietly disappears from AI search. I check this on every audit I run. See what your site's current configuration is actually doing.
Topics covered
Technical SEO
Ready to scale your organic growth?
Get a custom SEO strategy from Ilias Sami — trusted by agencies in Canada, Germany, UAE, and beyond.
