Should You Block GPTBot? Which AI Bots to Allow in robots.txt (Data From 51 Sites)
M. Zeeshan, Founder of GEOREX AI
Generative Engine & AI Search Intelligence
Share:
Blocking GPTBot stops OpenAI from using your pages for model training but has no effect on ChatGPT search results or user-requested page fetches, since GPTBot, OAI-SearchBot, and ChatGPT-User are three independent controls. Allow or block by type: block training crawlers like GPTBot, ClaudeBot, and Google-Extended if desired, but keep OAI-SearchBot, Claude-SearchBot, and PerplexityBot open for search visibility, and use separate robots.txt User-agent groups for each policy.
Key takeaways
GPTBot and OAI-SearchBot settings work independently, so blocking GPTBot does not remove a site from ChatGPT search answers.
In our review of 51 big sites, 18 blocked GPTBot while only 8 blocked OAI-SearchBot, and 10 of them blocked GPTBot but left OAI-SearchBot open.
Training-use controls include GPTBot, ClaudeBot and Google-Extended, while search and answer crawlers include OAI-SearchBot, Claude-SearchBot and PerplexityBot.
Robots.txt cannot reliably stop user-triggered fetchers like ChatGPT-User or Perplexity-User, since these may skip robots.txt entirely when a person asks for a specific page.
None of the 46 study brands we checked block any OpenAI, Anthropic, Perplexity or Google AI bot, yet ChatGPT named those brands in only 31% of real buyer questions. Open access alone does not get you mentioned.
Should you block GPTBot? What you gain and lose
Block GPTBot if you do not want OpenAI to use your site’s content for model training. Keep OAI-SearchBot allowed if you still want your pages eligible to appear in ChatGPT search.
M. Zeeshan, Founder of GEOREX AI
Author
Founder of GEOREX AI, an AI visibility platform that tracks how brands actually appear across ChatGPT, Claude, Perplexity, and Gemini, and turns the gaps it finds into a prioritized action plan.
Free AI visibility audit
See how AI engines describe your brand today
Check ChatGPT, Gemini and Google AI Overviews in minutes. No credit card needed.
OpenAI says GPTBot may crawl content for training its generative AI foundation models. Disallowing GPTBot signals that your content should not be used for that training (OpenAI).
That choice does not control ChatGPT search: GPTBot and OAI-SearchBot settings are independent. You can opt out of training while keeping search access, or block both.
In our own robots.txt review of 51 large sites, 18 blocked GPTBot and eight blocked OAI-SearchBot. Ten sites, including Reuters, Forbes, Bloomberg, Medium, eBay, Canva, Airbnb, Tripadvisor, LinkedIn and TechCrunch, blocked GPTBot while leaving OAI-SearchBot open (our review, 13 Oct 2026). That pattern shows how a site can separate training preferences from search visibility.
To opt out of training while keeping ChatGPT search access, use separate rules:
Allow OAI-SearchBot if you want your site eligible to appear in ChatGPT search answers. OpenAI says opting out removes sites from those answers, and recommends allowing the bot for search visibility (OpenAI).
Choose this setting based on whether ChatGPT search referrals and visibility matter to your site. Blocking OAI-SearchBot does not itself opt your content out of GPTBot training; set GPTBot separately.
ChatGPT-User: a user-triggered fetcher
ChatGPT-User fetches pages for certain actions initiated by users, rather than crawling automatically or signaling inclusion in search (OpenAI).
Blocking this agent may affect user-triggered page fetches where robots.txt rules apply, but OpenAI says those requests may not follow the rules. Do not treat this setting as a reliable way to control training or ChatGPT search; use GPTBot and OAI-SearchBot for those separate choices.
Which AI bots should you allow or block by type?
Choose access by separating training use, search visibility, and user-requested fetches.
Separate crawler policy by purpose, not by vendor name.
Bot type
What blocking costs or preserves
Decision signal
Training or model-use crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, Bytespider
Blocking GPTBot signals that your content should not be used for OpenAI model training; ClaudeBot may collect model-training data; Google-Extended controls Gemini model training and grounding, not Google Search inclusion or ranking (OpenAI, Anthropic, Google).
Block these when keeping your content out of model training matters more than anything you lose. Nothing here affects Google Search or ChatGPT search.
Search and answer crawlers: OAI-SearchBot, Claude-SearchBot, PerplexityBot
OpenAI says opting out of OAI-SearchBot removes sites from ChatGPT search answers. PerplexityBot surfaces and links websites in Perplexity results and is not used for foundation-model training (OpenAI, Anthropic, Perplexity).
Keep these allowed if you want to be found in AI search. Block them only if you want no presence in that app at all.
These fetch one page because a person asked for it. Perplexity says Perplexity-User generally ignores robots.txt, and OpenAI says ChatGPT-User requests may not apply robots.txt rules (OpenAI, Perplexity).
A robots.txt rule here is a request, not a block. If you need to stop these, use server or CDN level controls instead.
What robots.txt rules implement the three common policies?
Use separate User-agent groups to match your crawler policy, and keep Googlebot’s rules separate from AI-bot rules.
Keep Googlebot's rules in a separate group from AI-bot rules in every policy.
Policy 1: allow all
Choose this when you want the listed AI search bots to access your site. OpenAI says OAI-SearchBot can surface websites in ChatGPT search, and opting out removes sites from ChatGPT search answers; its setting is independent of GPTBot (OpenAI).
If no more specific rules restrict these agents, this catch-all allows them and other crawlers:
code snippet
User-agent: *
Allow: /
Check for specific groups elsewhere in the file that disallow individual bots. Leaving Claude-SearchBot and PerplexityBot unblocked keeps them eligible to surface your site in their search experiences (Anthropic; Perplexity).
Policy 2: block training crawlers, keep AI search open
Use separate rules for training crawlers and search bots when you want different access policies. This example blocks the listed training agents while leaving OAI-SearchBot, Claude-SearchBot, and PerplexityBot without a disallow rule:
Adjust the list to match your policy, and check the rest of the file for conflicting rules. Anthropic distinguishes ClaudeBot from Claude-SearchBot; blocking the search agent can reduce visibility in Claude search results (Anthropic).
Policy 3: block all the listed AI bots
Choose this when you do not want any of these AI companies to use your site, for training or for AI search. Know what it costs before you do it: OpenAI says opting out of OAI-SearchBot removes sites from ChatGPT search answers, and blocking Claude-SearchBot or PerplexityBot limits the same kind of visibility in those apps (OpenAI; Anthropic; Perplexity).
Keep Googlebot out of this list. Blocking Googlebot removes your pages from Google Search, and it does not stop Gemini training, which Google-Extended controls (Google). In our review of 97 readable robots.txt files, not one site blocked Googlebot.
What do 51 big websites block? Our robots.txt data
10 sites block GPTBot but keep ChatGPT search open
Ten of the 51 large sites block GPTBot while leaving OAI-SearchBot open. In the October 13, 2026 review, 18 sites blocked GPTBot, compared with eight that blocked OAI-SearchBot. The ten separating the two are Reuters, Forbes, Bloomberg, Medium, eBay, Canva, Airbnb, Tripadvisor, LinkedIn, and TechCrunch (our October 13, 2026 robots.txt review).
That pattern distinguishes training access from ChatGPT search access: a GPTBot rule does not necessarily mean a site blocks OAI-SearchBot. Eight sites blocked both, including The New York Times, BBC, CNN, Amazon, Quora, Yelp, IMDb, and Pinterest (our October 13, 2026 robots.txt review).
Blocking patterns vary across news and media sites
News and media sites in the sample blocked training-related crawlers more often than search and answer crawlers. Among 13 sites, all blocked ClaudeBot, Applebot-Extended, and Bytespider; 12 blocked CCBot, 11 PerplexityBot, nine Google-Extended, and eight GPTBot. Only three, The New York Times, BBC, and CNN, blocked OAI-SearchBot (our October 13, 2026 robots.txt review).
Across 97 readable files, Bytespider was blocked most often, by 26 sites. ClaudeBot and Applebot-Extended were each blocked by 23, CCBot by 22, and meta-externalagent by 20. GPTBot and PerplexityBot were each blocked by 18; Google-Extended by 15, Claude-SearchBot by 12, ChatGPT-User by 11, and OAI-SearchBot by eight (our October 13, 2026 robots.txt review). The Verge showed another distinction: it allowed GPTBot but blocked ChatGPT-User (same review).
When is robots.txt not enough to restrict AI requests?
Robots.txt only governs automated crawling. It cannot reliably stop a request that happens because a person asked an AI tool to open one specific page, since some of those fetches are built to skip robots.txt entirely.
Robots.txt governs crawling, not every AI-triggered page fetch.
Treat robots.txt as a crawler instruction, not a guaranteed request block
Separate the bot you're worried about into "crawler" or "user-triggered fetcher" before writing a rule. A disallow line only binds the first type, so the same bot name can behave two different ways depending on which mode made the request.
Check OpenAI's own documentation: ChatGPT-User fetches pages for certain user-initiated actions, is not an automatic web crawler or a Search inclusion signal, and may not apply robots.txt rules to those requests (OpenAI).
Check Perplexity's equivalent note: Perplexity-User fetches pages in response to user actions, is not used for training, and generally ignores robots.txt because the fetch is user-requested (Perplexity). This is distinct from PerplexityBot, the training and indexing crawler, which does read robots.txt.
Accept the practical result: a rule blocking PerplexityBot or GPTBot will not stop one reader from pasting your URL into a chat window and asking the model to read it live.
Picture a publisher that disallows GPTBot site-wide to stop training. A reader still pastes one article URL into ChatGPT and asks for a summary. That single fetch goes through ChatGPT-User, not GPTBot, so the crawl block never applies and the page gets read anyway.
Check provider-specific controls before relying on robots.txt alone
Some hosting and network platforms offer bot-management settings that sit outside robots.txt entirely, filtering requests at the server or edge level before they ever reach your pages. These controls can catch user-triggered fetchers that ignore robots.txt, but they vary by provider, so confirm the specific behavior in your own vendor's documentation before assuming any switch covers ChatGPT-User or Perplexity-User style requests.
What are the most common robots.txt mistakes with AI bots?
The main mistakes are blocking search access when you only meant to stop training, confusing Google-Extended with Googlebot, and using one catch-all rule for every crawler. Set rules for the specific bots you want to control, then check that the other crawlers are still allowed.
Blocking Googlebot or using one catch-all rule for every bot
Do not block Googlebot or apply a blanket rule to every crawler. Google says Google-Extended controls whether content may help train future Gemini models; it does not affect inclusion in Google Search or act as a Search ranking signal (Google’s crawling guidance). Blocking Googlebot is not a way to control that training use. Name the specific AI bots you want to block, then review the file for rules that could affect other crawlers.
Blocking OAI-SearchBot when you only meant to stop training
Give GPTBot and OAI-SearchBot separate rules if you want to block training-related access while keeping search access. In our audit of 13 news and media sites, only New York Times, BBC, and CNN blocked OAI-SearchBot. That contrast shows why a rule aimed at one bot should not automatically cover another. Check the deployed file to confirm each bot receives the intended rule.
Confusing training controls with search controls
Treat similarly named bots as separate crawlers with different purposes. Anthropic distinguishes ClaudeBot, which collects content that could contribute to model training, from Claude-SearchBot, which supports search; blocking either separately limits its stated purpose, and Anthropic says its bots honor robots.txt directives (Anthropic’s crawler guidance). Apply the same care to Google-Extended and Googlebot: blocking the former does not block Google Search, and blocking the latter is not a substitute. After changing rules, verify that the intended bots are blocked and the crawlers you want to allow remain permitted.
How should you choose a policy and measure its effect?
The free AI Crawler Checker reads a robots.txt file and shows the status of each AI bot.
Choose the access policy that matches your objective
Choose crawler access based on whether your priority is limiting training use, appearing in AI search, or both.
If you want to limit training use while preserving search visibility, block training crawlers selectively and leave search crawlers available. GPTBot and OAI-SearchBot settings are independent; opting out of OAI-SearchBot removes sites from ChatGPT search answers (OpenAI’s crawler guidance). Google-Extended affects Gemini use, not Google Search inclusion or ranking (Google’s crawler documentation).
If you want AI search visibility, allow relevant search and answer crawlers, then check results by platform. Perplexity says PerplexityBot surfaces and links websites in search results, and is not used to crawl content for foundation models (Perplexity’s crawler guidance). Treat user-triggered fetchers such as Perplexity-User separately: verify their requests and your access controls rather than assuming a training-crawler rule governs them.
Treat our 13 October 2026 review as a benchmark, not a target. None of the 46 study brands blocked the listed OpenAI, Anthropic, Perplexity or Google AI bots, yet ChatGPT named those brands in only 31% of real buyer questions. Open access alone does not ensure mentions.
Measure mentions and citations separately
Measure brand mentions and direct citations separately, then compare changes with crawler access and query-level results.
Before changing rules, record a baseline for the same buyer questions across the AI platforms you track. GEOREX monitors AI visibility across ChatGPT, Claude, Gemini, and Perplexity, and separates brand mention rate from direct-citation rate.
After a change, rerun the same questions and compare mention and citation rates by platform. Also check whether cited pages are accessible and indexed; Google says AI Overviews and AI Mode links require pages to be indexed and eligible for Google Search (Google’s AI-features guidance).
Set a review window and a decision threshold before testing. If access changes but mentions and citations do not, investigate content relevance and page eligibility before changing crawler rules again.
Yes, OAI-SearchBot can stay allowed while GPTBot is blocked. OpenAI manages the two crawlers independently, so disallowing GPTBot only opts the site out of training use and has no effect on ChatGPT search inclusion, which depends on OAI-SearchBot remaining open (OpenAI).
