What Is llms.txt and Does It Actually Work? (Data From 100 Sites)

The format has one required element: an H1 title naming the site or project. A summary blockquote, introductory details, and H2 sections containing Markdown links are optional. A final section for less relevant links is conventionally titled “Optional” (format details).

Keep the file selective and maintain its links. A short list of useful, current pages is easier to scan than a long directory of every site URL. Group links under clear H2 headings when the site has distinct topics. Include introductory text only when it helps explain the project or how readers should use the linked material.

llms.txt vs robots.txt vs llms-full.txt: What Is the Difference?

llms.txt, llms-full.txt, and robots.txt solve three different problems, and only one of them is an actual internet standard. llms.txt is a short Markdown guide meant to point AI agents toward a site's most useful pages (llmstxt.org). llms-full.txt is a separate, much larger file that packs full page content into one document, a format Mintlify uses to bundle an entire documentation site's Markdown into a single resource (Mintlify). robots.txt is the only one of the three backed by a formal internet standard, RFC 9309, which governs how crawlers should read crawl permissions (RFC 9309).

The practical difference comes down to size and intent. llms.txt stays short on purpose: a required H1 title, an optional blockquote summary, then H2 sections of Markdown links, sometimes ending in an "Optional" section for lower-priority pages (llmstxt.org). llms-full.txt drops that brevity rule and just concatenates full page content, so it can run into thousands of lines on a documentation-heavy site (Mintlify). robots.txt does neither: it is not a content guide at all, just a set of allow and disallow rules for specific crawlers (RFC 9309).

Here is how the three files compare on purpose, format, and standard status.

Before trusting any llms.txt file, test it the way the original proposal suggests: give an agent only that file as a starting point and ask it real questions about the site (llmstxt.org). For a closer look at how robots.txt rules affect AI crawlers specifically, see this guide on whether ChatGPT can crawl your site.

Does llms.txt Actually Work? What the AI Companies Have Said

No major AI company has confirmed that it reads llms.txt, and the one company that has addressed it directly says it makes no difference. Google Search Central states plainly that Google Search does not use llms.txt, and having one on a site neither helps nor hurts rankings or visibility in Google's AI features (Google Search Central). That single sentence settles the question for anyone optimizing specifically for Google.

OpenAI's public crawler documentation tells a similar story by omission. OpenAI documents OAI-SearchBot as the crawler that surfaces websites in ChatGPT search results, and instructs site owners to manage that bot's access through robots.txt, not llms.txt (OpenAI). OpenAI's guide covers GPTBot and other named crawlers in the same detail, and nowhere does it describe a process for fetching or parsing an llms.txt file. If OpenAI's systems read llms.txt today, that behavior is not documented anywhere a site owner can verify.

OpenAI does give a clear, actionable instruction, just not about llms.txt: allow OAI-SearchBot in robots.txt if you want a chance at appearing in ChatGPT search results (OpenAI). That is the lever OpenAI confirms exists. llms.txt is not part of that lever based on what the company has published.

We found no public statement from Anthropic, Perplexity or the team behind the Gemini app saying their systems read llms.txt. A missing statement is not a denial, but it does mean nobody can show you proof that the file changes what these assistants cite or recommend.

Here is the practical read: llms.txt is a voluntary convention, not a recognized crawling standard any major model provider has committed to following. Robots.txt rules for GPTBot and OAI-SearchBot are the documented, verifiable way to control what OpenAI's systems can access. Treat llms.txt as a low-cost, unproven extra, not a replacement for getting robots.txt and site structure right. For what does move AI answers, see our guide to generative engine optimization.

Our Data: How Many Sites Use llms.txt

We checked this ourselves. On 14 October 2026 we looked for an llms.txt file on the same 100 websites we used in our AI crawler check: 49 brands from our AI brand visibility study and 51 well-known sites such as Amazon, Shopify, the BBC and GitHub. A site only counted as having the file if /llms.txt returned a real text or Markdown file. A 404 page, a normal HTML page or a redirect to the homepage did not count. 7 sites blocked the check with a bot wall, so the numbers below cover 93 sites.

• 37 of 93 sites (40%) have an llms.txt file: That is more than most people expect, but it is still well under half.
• Software companies lead: Shopify, HubSpot, Salesforce, Notion, Slack, Atlassian, Mailchimp, Dropbox, Adobe and GitHub all have one. Zoom, Canva, Microsoft and Apple do not.
• Big retailers mostly do not: Target and Etsy have one. Amazon, Walmart, eBay, IKEA, Best Buy, Nike, Zara and H&M do not.
• News sites almost never do: Bloomberg was the only one of 10 readable news and media sites with a file.
• Study brands: 23 of 48 (48%): , from Gorgias and Pipedrive to Liberty Mutual and EDF Energy.

The more useful question is whether the file changes anything. For the 48 study brands we already knew how often ChatGPT and Gemini named each one across 10 real buyer questions, so we split them by whether they have an llms.txt:

Brands with the file were named a little more often, but a gap of 2 to 4 points across 48 brands is small enough to be noise. The brands that add llms.txt also tend to be the ones that invest more in documentation and content in general, so the file itself gets little credit. Our read: llms.txt is not what decides whether ChatGPT or Gemini recommends you. What you publish, and where other sites mention you, matters far more.

You can run the same check on your own site in a few seconds. Open yoursite.com/llms.txt in a browser. A real file shows plain text that starts with a # title. In our check, several big sites, including Netflix and Spotify, answered /llms.txt with a redirect to a normal web page. That is not an llms.txt file, even though the browser shows something.

How to Create and Add an llms.txt File to Your Website

Creating an llms.txt file means writing a short Markdown document in a set format, then placing it in your site's root directory. The steps below cover drafting, formatting, and publishing, based on the format described by llmstxt.org.

1. Pick the scope: Decide if the file covers your entire domain or just one subpath. The format allows either: a file at the root applies site-wide, while a file placed at a subpath covers only the pages beneath that location (llmstxt.org).
2. Write the required H1 title: Every valid llms.txt file starts with a single H1 heading naming the site or project. This part is not optional. Skipping it, or using more than one H1, breaks the expected format (llmstxt.org).
3. Add a blockquote summary: Directly under the H1, include a short blockquote describing what the site or business does. Follow it with a few plain sentences of background or guidance for anyone reading the file (llmstxt.org).
4. Build H2 sections with Markdown links: Group useful pages under H2 headings, such as "Documentation" or "Products," with each item written as a Markdown link plus a short description. An optional final section, often titled "Optional," can hold lower-priority links (llmstxt.org).
5. Check the formatting before publishing: Open the raw file and confirm the Markdown renders cleanly: one H1, a blockquote, then H2 sections with links. This check only confirms the syntax parses correctly; it does not confirm that any AI company actually fetches the file live.
6. Upload the file to your site root: (yoursite.com/llms.txt) using your CMS file manager, FTP client, or hosting dashboard, the same way you would publish a robots.txt file.
7. Use a generator if you want to skip manual formatting: The free GEOREX AI llms.txt Generator creates a formatted file for your site that you can review, edit and then upload.

Once the file is live, the most reliable test is to have an agent read it directly and report back which links it sees, rather than relying on the formatting check alone.

A Real llms.txt Example

A working llms.txt file follows a simple Markdown pattern: one title, a short summary, then grouped links. Below is a plain example for a fictional project management software company, built to match the format defined in the /llms.txt spec.

This structure matches the required format: an H1 title first, an optional blockquote summary, then H2 sections holding Markdown links, with an "Optional" section reserved for lower-priority pages (llmstxt.org). The file can sit at the site root or under a subpath, and it only applies to URLs beneath that path (llmstxt.org). Keep descriptions short and factual. Link only to pages that explain what the business does, how pricing works, and where documentation lives, since that is the content an agent is meant to find useful.

Common Mistakes and What llms.txt Will Not Do

llms.txt will not control crawler access, improve Google rankings, or guarantee that any AI platform reads it, and most mistakes come from treating it like a rule file instead of a reference document. Check your own file against this list:

• Mistake: using llms.txt to block or allow crawlers: Access control belongs to robots.txt, which is defined by RFC 9309 and tells crawlers which paths they are requested to access or avoid (RFC 9309).
• Mistake: assuming robots.txt rules are enforceable permissions: RFC 9309 states plainly that these rules are not a form of access authorization (RFC 9309), so any blocking still relies on the crawler choosing to comply.
• Mistake: expecting an SEO lift: Google's own generative-AI guide says Google Search does not use llms.txt and that having one neither helps nor hurts Search visibility (Google Search Central).
• Mistake: assuming ChatGPT reads it automatically: OpenAI's crawler documentation covers OAI-SearchBot and robots.txt management but does not describe any llms.txt parsing (OpenAI).
• Mistake: skipping robots.txt setup: An llms.txt file does nothing to manage AI crawler access on its own. Check what your robots.txt actually allows with the free AI Crawler Checker.

Treat llms.txt as an optional, unverified reference file, not a ranking lever or an access-control file.

Sources

1. The /llms.txt file, v2, llmstxt.org (2024-09-03)
2. llms.txt, Mintlify
3. RFC 9309: Robots Exclusion Protocol, RFC Editor / IETF (2022-09)
4. Google's Guide to Optimizing for Generative AI Features on Google Search, Google Search Central
5. Overview of OpenAI Crawlers, OpenAI
6. AI consumability, Cloudflare (2026)

We Asked ChatGPT and Gemini 500 Buyer Questions. Their Own Top-10 Brands Were Missing Most of the Time

We took 50 brands that ChatGPT and Gemini rank in their own top-10 lists and asked both apps the questions real buyers type. Most of the time, the brand was not named.