Loading
Loading
Loading
Free AI crawler checker
Enter your site. We read your real robots.txt and page tags and show which AI search crawlers and training crawlers are allowed, which are blocked, and the exact line that decides it.
Free, no account. Add a path (yourbrand.com/blog) to check one page.
Three kinds of AI crawler
Most robots.txt mistakes come from treating all AI bots the same. Blocking a training crawler is fine; blocking a search crawler hides you from AI answers.
OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot
They build the index an AI assistant searches when it answers. Block one and that assistant can't find or cite your pages. Keep these allowed.
ChatGPT-User, Claude-User, Perplexity-User
They open a page right now because a person asked the assistant to. Some of them say they do not follow robots.txt for this, because a human made the request.
GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended
They collect pages to train future models. Blocking them is a normal choice and does not remove you from AI answers. Google-Extended is only a control word: Googlebot still reads your site.
Our data, 13 Oct 2026
We read the robots.txt of 100 well-known websites (97 could be read): 49 brands from our AI visibility study and 51 big sites such as Amazon, the New York Times and LinkedIn. Brands almost never block AI crawlers. Big publishers block training bots far more often than search bots.
0 of 46
brands from our AI visibility study block GPTBot or OAI-SearchBot
18 of 51
big websites block GPTBot, OpenAI's training crawler
8 of 51
big websites block OAI-SearchBot, the crawler behind ChatGPT search
0 of 97
readable robots.txt files block Googlebot
Blocked means the rules disallow the homepage for that bot. Checked in a normal browser on 13 October 2026; files change, so check your own site above.
Read the full study: how to check if ChatGPT can crawl your website
Crawler reference
Use the exact name in the first column in your robots.txt. Keep the search and live-fetch crawlers allowed if you want to show up in AI answers. Blocking the training crawlers is your choice.
| User-agent | Company | What it does | Our advice |
|---|---|---|---|
| Googlebot | Google Search, AI Overviews and AI Mode | Keep allowed | |
| Bingbot | Microsoft | Bing and Copilot answers | Keep allowed |
| OAI-SearchBot | OpenAI | ChatGPT search | Keep allowed |
| ChatGPT-User | OpenAI | Opens a page when a ChatGPT user asks | Keep allowed |
| Claude-SearchBot | Anthropic | Claude web search | Keep allowed |
| Claude-User | Anthropic | Opens a page when a Claude user asks | Keep allowed |
| PerplexityBot | Perplexity | Perplexity answers | Keep allowed |
| Perplexity-User | Perplexity | Opens a page when a Perplexity user asks | Keep allowed |
| GPTBot | OpenAI | Training OpenAI models | Your choice |
| ClaudeBot | Anthropic | Training Claude models | Your choice |
| Google-Extended | Gemini training and grounding (a control word, not a crawler) | Your choice | |
| Applebot-Extended | Apple | Apple AI training (a control word, not a crawler) | Your choice |
| CCBot | Common Crawl | Open web dataset many models train on | Your choice |
| Meta-ExternalAgent | Meta | Training Meta AI models | Your choice |
| Bytespider | ByteDance | Training ByteDance models | Your choice |
Example robots.txt
This is the setup most businesses want. Training crawlers are blocked in one group. Every other crawler, including Googlebot and the AI search crawlers, falls under the * group and stays allowed.
robots.txt
# Block AI training crawlers User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Meta-ExternalAgent User-agent: Bytespider Disallow: / # Everyone else, including Googlebot, Bingbot, # OAI-SearchBot, Claude-SearchBot and PerplexityBot User-agent: * Allow: /
What it checks
Nothing is guessed. If a file can't be read, the result says so. The crawler list comes from the AI companies' own crawler pages.
Both belong to OpenAI but do different jobs. GPTBot collects pages to train OpenAI's models. OAI-SearchBot builds the index ChatGPT search uses when it answers and cites sources. You can block one and allow the other, and many big sites do exactly that: in our check of 51 big websites, 18 block GPTBot but only 8 block OAI-SearchBot.
No. GPTBot only collects pages to train OpenAI's models. ChatGPT search uses OAI-SearchBot, and ChatGPT-User opens a page when a person asks. If you block GPTBot but allow OAI-SearchBot, ChatGPT can still find and cite you.
No. Google-Extended is a robots.txt control word, not a crawler, and it covers Gemini training and grounding. AI Overviews and AI Mode use pages that Googlebot crawls for Google Search. Blocking Googlebot would remove you from Search and AI Overviews together.
robots.txt is only a request. Your CDN or firewall (for example Cloudflare's AI bot setting) can block AI crawlers before they ever reach robots.txt, and this check can't see that. Look at your CDN's bot settings and its logs for blocked AI user agents.
A missing file (404) means every crawler may read everything. A server error (5xx) is different: under RFC 9309 crawlers treat it as 'do not crawl anything' until it works again, so a broken robots.txt can hide your whole site.
Put the training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and others) in one group with Disallow: /, and leave a User-agent: * group that allows everything else. The example on this page does exactly that. Then run this check again to confirm OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot still show as allowed.
No. On 13 October 2026 we read the robots.txt of 100 well-known websites (97 could be read). Only 32 of them name any AI bot at all. None of the 46 brands from our AI visibility study blocks an OpenAI, Anthropic, Perplexity or Google AI crawler. Big publishers are the exception: all 13 news sites we checked block ClaudeBot, but only 3 block ChatGPT search.
Usually about a day. Google says it generally caches robots.txt for up to 24 hours, and OpenAI says a change can take about 24 hours to reach its search crawler. Run this check right after you upload the new file to confirm the live file says what you meant.
Yes, it is free and needs no account. We read only public files (robots.txt, the page you enter, llms.txt and sitemap.xml) and keep a short log of which site was checked so we can see how the tool is used.
No. Being readable is the first step. Whether ChatGPT, Gemini or Perplexity actually names your brand depends on what is written about you across the web. The free GEOREX AI visibility audit checks that with real buyer questions.
Free, no credit card
Reading your site is step one. The free AI visibility audit asks ChatGPT real buyer questions and shows whether it names you or a competitor.
Run the free AI visibility audit