Loading
Loading
Free AI crawler and llms.txt tools, plus simple technical guides on robots.txt, AI bots and making your site readable for ChatGPT, Gemini and Perplexity.
What llms.txt is, how it differs from robots.txt, what Google and OpenAI have said about it, and our check of 100 websites: 37 of 93 have the file, but it barely changes how often ChatGPT names a brand.
We checked the robots.txt of 100 websites. Here is how to see if ChatGPT can crawl your site in five minutes, and what the big sites block.