llms.txt: What It Is and Whether Your Site Needs One

The short answer: llms.txt is a small Markdown file at your site's root that hands AI assistants a clean, curated map of your content — what the site is and which pages matter. It will not lift your Google rankings, it is not a formal web standard, and no major AI company has committed to reading it systematically. Whether you need one comes down to a simple question: do AI assistants ever need to understand your site? For documentation sites and dev tools, the answer is yes — it's ten minutes of cheap insurance. For a five-page brochure site, you can skip it. The rest of this post is the honest evidence behind that verdict.

What llms.txt actually is

Web pages are built for people. Every paragraph of real content arrives wrapped in navigation menus, cookie banners, JavaScript, and layout — and an AI assistant trying to answer a question about your site has to dig through all of it, often inside a tight context window where every wasted token costs time and money. The llms.txt proposal, published by Jeremy Howard (of fast.ai and Answer.AI) in September 2024, offers a simple fix: one Markdown file at /llms.txt that states what the site is and links straight to the pages worth reading, each with a one-line note on what it covers.

The format is deliberately minimal. An H1 with the site name (the only required part), a blockquote with a short summary, then H2 sections of links written as - [Name](https://url): notes. A section called "Optional" holds secondary links. That is nearly the whole spec — it fits on one page, which is rather the point. The proposal is now at v2 (updated August 2026), which also suggests serving clean Markdown versions of important pages alongside the index, but the core file remains the same small map.

Think of it as the file you would write for a smart new colleague: not everything the company knows, but the five pages they'd need on day one, with a sentence on each explaining why.

What it is not

Three misconceptions do most of the damage here, so let's clear them up front.

It is not robots.txt. robots.txt controls access — it tells crawlers which URLs they may fetch. llms.txt controls nothing. It doesn't allow or block anything, and it grants no permissions. If you want AI crawlers kept out of parts of your site, that still happens in robots.txt, full stop.

It is not a ranking signal. This is the one vendors selling "AI SEO" packages least want you to internalize: llms.txt does not improve your Google rankings, and Google says so directly. Google's Search Central guidance on AI features and your website states there are no additional technical requirements for appearing in AI Overviews or AI Mode beyond standard indexing eligibility — and that robots.txt directives remain the control for managing how your content is crawled. No machine-readable file is needed, and none is rewarded.

It is not a formal standard. No standards body has ratified llms.txt. It is a community proposal, developed in the open, with real momentum — documentation platforms like Mintlify and GitBook generate one automatically, WordPress plugins like Yoast SEO and AIOSEO now offer generation, and Chrome's Lighthouse even audits for one as part of its agentic browsing checks. Momentum is not the same as ratification, and the distinction matters when someone tries to sell you urgency.

Who actually reads it? The honest adoption picture

Here is where most articles about llms.txt either hype or doom-monger. The evidence, as of late 2026, is genuinely mixed — and worth stating plainly:

Two honest caveats. First, independent measurement is thin: one widely cited 180-day study of ten sites found only two saw a measurable AI-traffic lift after adding the file. Second, the only measurement you can fully trust is your own — log requests to /llms.txt by user agent for a month and you'll know exactly which crawlers fetch yours. Everything else is someone else's anecdote.

The pattern in the data is consistent: llms.txt is genuinely useful where an AI agent needs to work with your content (documentation, APIs, references), and speculative where you hope an AI will cite you more. Keep that split in mind — it drives the entire decision below.

Do you need one? A decision framework

Forget the hype cycle. Run your site through this table:

Notice what the "yes" rows have in common: someone — a developer, an agent, a researcher — regularly needs to understand the site rather than just find it. Search engines find things; llms.txt explains them. If nobody needs your site explained, the file is decoration.

One more consideration: staleness. An llms.txt that nobody owns rots fast — new pages never get added, dead links accumulate, and a stale map is worse than none because it misdirects the very agents it was meant to help. If you publish one, put a quarterly reminder next to whatever owns your sitemap. If you can't commit to that, don't publish it.

llms.txt vs robots.txt vs sitemap.xml

These three root files get confused constantly because they live in the same neighborhood. They do completely different jobs:

The practical takeaway: these files complement each other rather than compete. A healthy site can run all three — robots.txt guarding the doors, sitemap.xml listing the rooms, llms.txt giving the tour. Our robots.txt and sitemap guide covers the first two in depth, including the classic mistakes (like blocking a crawl when you meant to block indexing) that quietly cost sites traffic.

llms.txt for WordPress and other CMS sites

If the decision table above put you in the "yes" camp, the good news is you may not need to hand-write anything. Several platforms now generate the file for you: Yoast SEO and AIOSEO both ship llms.txt generation for WordPress, Wix generates one for every site on its platform, and documentation hosts like Mintlify and GitBook create one automatically. If you use one of these, check what it generated before assuming it's good — auto-generated files tend to include everything, which defeats the purpose of curation. Trim it to the pages that actually answer questions.

For a hand-built file, or to check what your CMS produced, a dedicated llms.txt generator builds a spec-valid file from your sitemap or a simple form and validates the result — including the things that actually break the file in practice, like relative links or a server quietly serving an HTML 404 page instead of text. Ten minutes, once, then a quarterly glance to keep it fresh.

Adding one takes ten minutes

If you've decided the file earns its place: write the H1 and summary, list your best pages with one-line descriptions each, keep it under a few dozen links, upload it to your site root so it loads at yoursite.com/llms.txt, and validate that the server returns it as plain text rather than an HTML error page. Then add that quarterly review reminder. That's the whole project — which is exactly why the "should I?" question matters more than the "how?" one. The file is trivial to create; the judgment call is whether anyone will ever read it.

Frequently asked questions

Does llms.txt improve Google rankings?
No. Google's own guidance states there are no additional technical requirements — no special files — for appearing in Search, AI Overviews, or AI Mode beyond standard indexing eligibility. llms.txt is invisible to Google's ranking systems. Anyone selling it as an SEO lever is selling something Google says doesn't exist.

Do ChatGPT or Claude actually read llms.txt?
Partially. Anthropic and Perplexity say their retrieval systems read the file; OpenAI's GPTBot has been observed fetching it but OpenAI hasn't committed to using it; Google has no plans to support it. AI coding assistants (Cursor, Claude Code, Continue) are the most consistent real-world consumers today. The honest summary: useful for agents working with your docs, unproven for everything else.

What's the difference between llms.txt and robots.txt?
robots.txt controls access — it tells crawlers which URLs they may fetch, and it's where you block AI crawlers if you want them out. llms.txt controls nothing; it's a curated reading guide for AI assistants. They complement each other: one guards the doors, the other gives the tour.

What is llms-full.txt?
A companion convention some documentation sites use: a single file containing the full text of their docs, rather than just links to them. It is not part of the core llms.txt proposal — the spec site doesn't define that filename — so treat it as an optional extra some tools understand, not a requirement.

What is the best llms.txt generator?
The best llms.txt generator tools validate against the actual spec (H1 present, proper link format, absolute URLs), flag a file served as HTML instead of text, and don't require a signup for a ten-minute job. Generate from your sitemap if you have one, then curate the result down — an unedited sitemap dump misses the entire point of the file.

How do I know if anything actually reads my llms.txt?
Check your own server logs. Log requests to /llms.txt separately and group by user agent — within a month you'll see exactly which crawlers fetch it (look for GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, and similar tokens). It's the only measurement that isn't someone else's anecdote, and it settles the "does anyone read it" question for your domain specifically.