What Is llms.txt and Does Your Business Actually Need One?

If you have talked to anyone about AI search in the last year, someone has told you that you need an llms.txt file. It gets described as “the new robots.txt,” which sounds important and is not accurate.

The honest answer for most businesses is that you probably do not need one, that it will not hurt you, and that it takes about twenty minutes if you want it anyway. Here is the reasoning, because the reasoning is more useful than the verdict.

What it actually is

llms.txt is a plain Markdown file that sits at the root of your domain, at yoursite.com/llms.txt. It contains a curated list of your most important pages with a short description of each one. Think of it as a table of contents written for a machine rather than a person.

It was proposed by Jeremy Howard in September 2024. The specification is short and readable in a few minutes.

The idea is reasonable. A language model reading your site has to parse navigation, cookie banners, footers, and markup to find the actual content. A curated Markdown index skips all of that and points straight at what matters. For a documentation-heavy site, that saves a lot of wasted processing.

What it is not

Three clarifications, because all three get muddled constantly.

It is not robots.txt. robots.txt controls access and tells crawlers what they may not fetch. The major AI crawler operators document their compliance with it. llms.txt controls nothing. It cannot block a crawler, restrict training use, or prevent scraping. The two files do opposite jobs and only one of them has teeth.

It is not a standard. It has no backing from the W3C, the IETF, or any standards body. It is a community convention, which means adoption is voluntary and inconsistent by design.

It is not a ranking factor. This is the part being oversold hardest.

Who actually reads it

This is where the evidence matters more than the theory.

Google has said no, on the record. Google representatives confirmed in 2025 that Google Search does not support llms.txt and has no plans to, with one comparing it to the long-discredited keywords meta tag. Google has been consistent that AI Overviews and AI Mode rely on ordinary SEO signals rather than special markup.

The major providers have not committed to it. As of 2026, no major AI company has publicly stated that its production systems read or act on llms.txt from arbitrary sites when generating answers.

The log data is unflattering. Large-scale studies of crawler behavior have found that the retrieval bots which actually feed AI answers request llms.txt very rarely. One analysis noted that AI bots never probed for the file on domains where it did not exist, which is telling. If a crawler wanted the file, cheap speculative probing is exactly what you would expect to see. A meaningful share of the traffic that llms.txt files do receive comes from SEO audit tools checking whether the file exists, which is the industry measuring itself.

Adoption is low. Estimates range from roughly 2 percent of sites to around 10 percent depending on the sample, concentrated heavily in software documentation, developer tooling, and open-source projects. Marketing and local business sites have barely adopted it, and the use case there is weak.

The case that it is not useless

There is a real counter-argument and it deserves fair treatment.

Several major AI companies publish llms.txt files for their own documentation. Anthropic has recommended the convention in its guidance on writing for agents, and OpenAI maintains them for some of its developer projects. Perplexity has been described as retrieving the file to help prioritize pages. Chrome’s Lighthouse added an agentic browsing audit in 2026 that checks whether a site provides one.

The reconciliation between “no crawler consumes it” and “every major AI lab publishes one” is that llms.txt serves agentic retrieval rather than conversational search. When a developer asks a coding assistant to work against a company’s documentation, the agent can fetch that llms.txt as a curated index and traverse only the pages it needs rather than scraping the whole site. Coding agents like Cursor and Claude Code do read it when a user points them at a domain.

That is a real use case. It just has nothing to do with whether a homeowner searching for a plumber gets your name.

So should you make one?

A straightforward test.

Worth doing if:

• You publish technical documentation that developers point AI coding assistants at.
• You run internal retrieval systems or an MCP server against your own content and want a controlled entry point.
• You have a large content library, a spare afternoon, and you want the cheap option on a convention that might gain traction.

Not worth doing if:

• You are a local service business, a professional practice, a restaurant, or a retailer. The audience the file serves is not your audience.
• Your team has limited capacity and there is unfinished work on your Google Business Profile, your reviews, your site structure, or your page content. All of those have measurable effect. This does not.
• Someone is quoting you a real price for it as an AI visibility service. That is the clearest signal to walk away.

If you do it, do it correctly

The format is a Markdown file with an H1 for your site name, a short blockquote summary, and then H2 sections containing linked lists of pages with one-line descriptions each. Keep it to the pages that genuinely matter and keep it current, because a stale index is worse than none.

Two mistakes to avoid.

Do not generate Markdown copies of every page on your site. A popular implementation approach creates a parallel .md version of each page. If those files are indexable, you have just created duplicate content at scale, which dilutes crawl budget and can suppress the original pages. If you do this, make sure the copies are not indexable.

Think carefully about llms-full.txt. The companion format concatenates your entire content into one file. That is convenient for agents and it is also a clean, complete scrape handed to anyone who asks. If your business depends on people visiting your pages, consider whether that trade is one you want.

The broader point

llms.txt is a useful case study in how AI search advice is currently being sold. A plausible idea with a narrow real application gets repackaged as an urgent competitive necessity, and businesses spend attention on it that would produce far more return somewhere else.

The things that verifiably affect whether AI engines describe your business accurately are unglamorous: consistent business information, a complete and active Google Business Profile, recent reviews, pages that answer questions directly, and credible third-party mentions. We covered those in more detail in our guide to local SEO and search visibility.

None of that is as interesting as a new file format. All of it works.

If someone has told you that you need an llms.txt file and you are not sure whether that is advice or a sales pitch, ask us. We will tell you honestly, including when the answer is that you do not need us for it.