What llms.txt does, what it does not do, and who actually reads it

The llms.txt proposal is simple and appealing: a markdown file at the root of your site listing what you publish, so a language model reading you does not have to reconstruct your structure from navigation menus. It costs an afternoon to write. Adoption is rising steadily.

It is also, on current evidence, almost entirely ignored by the crawlers it was designed for.

What the logs show

Large-scale log analysis finds that 97% of published llms.txt files receive zero requests. GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and Google-Extended overwhelmingly skip the file and crawl HTML directly.

The vendors have not committed to it either. As of the first quarter of 2026, no major AI company has publicly committed to reading or acting on llms.txt in production. Google’s June 2026 documentation update went further and stated plainly that the file has no effect — positive or negative — on Search rankings or AI Overviews.

Adoption, meanwhile, keeps climbing. Depending on the panel you measure, it runs from about 10% across a 300,000-domain sample to over half of a small fixed panel of tech-forward hosts. The gap between those numbers is itself the finding: llms.txt has been adopted enthusiastically by the kind of site that reads posts like this one, and ignored by everything else.

Who does read it

Not nobody. The file’s demonstrable consumers today are agentic browsers and coding tools — the assistants that fetch a documentation site while writing code against it. That is why developer-documentation sites adopted it far faster than anyone else: they have a consumer that actually reads it.

If your audience writes software against your product, llms.txt is doing work. If your audience is a buyer asking ChatGPT which vendor to shortlist, it is not.

Why this site ships one anyway

We publish /llms.txt and /llms-full.txt, and they are generated from the content rather than hand-maintained. Given the evidence above, that deserves an explanation rather than a shrug.

Three reasons, in descending order of honesty:

It costs nothing to keep correct. Ours is generated at build time and every URL in it is verified before the build is allowed to finish. A hand-maintained file that drifts is worse than no file; a generated one is close to free.

The consumer set is growing, not shrinking. Coding agents read it today. Whether answer engines follow is unknown, and the cost of being early is one build step.

It is a legible statement of structure. Writing the file forces you to say what your publication actually contains. That exercise has value even if no crawler ever fetches the result.

What we do not claim is that it improves visibility in AI answers. Nothing in the current evidence supports that, and a publication about accurate description should not make a claim it cannot support about its own tooling.

What to do instead

If the goal is being described accurately by an assistant, spend the afternoon on the things the crawlers demonstrably do read: the HTML itself, served without requiring JavaScript, and the third-party pages that describe you. llms.txt is a reasonable ten minutes at the end of that work. It is not the work.