llms.txt: What It Is and Whether It Does Anything

llms.txt has moved faster through technical SEO checklists than through engine documentation. The file is real. The search case is still unproven.

The llms.txt specification proposes a clean Markdown map of your site so an AI agent does not have to extract the same information from HTML. Its v2 update says thousands of sites now publish one and that documentation platforms generate them automatically. That proves sites are adopting it. It does not prove engines read it.

Google is the only major provider to give a direct answer. Its guide to generative AI features in Search says Google Search does not use llms.txt and that publishing one will neither help nor harm visibility. The other major providers have published no support either way.

The question is not whether you can add the file. It is who will use it, and for what. This guide separates adoption from evidence, then explains when llms.txt is still worth adding.

Key Takeaways

• The specification is real and well designed: a markdown index at a known path, plus markdown versions of pages, plus standard link relations so an agent can find both.
• Google has published a direct no: its own guide states that Google Search does not use these files, which covers AI Overviews and AI Mode.
• Publishing is not reading: OpenAI, Anthropic, and Gemini publish llms.txt files for their own developer docs. That says nothing about whether their engines fetch yours.
• The strongest case is coding agents, not search: these files earn their keep on documentation sites, where an agent needs an API reference and nobody wants to parse HTML to find it.
• Treat it as cheap hygiene, not a lever: take the generated version if your platform makes one, and spend the afternoon on something an engine has documented.

What llms.txt Is

llms.txt is a proposed Markdown index that you publish on your site. It provides a short description of the site and a curated list of pages an AI agent may need. The conventional location is /llms.txt, which gives agents a predictable place to look without guaranteeing that any particular engine will use it.

The filename includes the dot, although searches for “llms txt” refer to the same file.

The problem it solves is token cost. The spec says an HTML page wraps its information in navigation, ads, and JavaScript. Converting that back into clean text is difficult and imprecise. A markdown index skips the work.

What the Specification Proposes

The proposal has three main parts.

• A file at a known path: /llms.txt at the site root, or at any subpath, covering the URLs under that path. Where more than one file applies, an agent should use the most specific.
• Markdown versions of pages: served at the same URL as the original, either with .md appended (page.html.md) or with the extension replaced (page.md).
• Standard link relations for discovery: rel="alternate" type="text/markdown" points at a page's markdown version, and rel="describedby" points at the llms.txt file that covers it.

v2 added those link relations. The changes log says discoverability was the commonest request after v1. The link relations address a basic discovery problem: how an agent can find a page’s Markdown version without guessing the URL.

Both forms are allowed. You can ship the relations as an HTML <link> element or as an HTTP Link: response header. The HTTP header is especially useful because it can be configured at the server or CDN level, requires no changes to individual pages, and can cover non-HTML resources.

Which Engines Have Said They Read It

One major engine has published an answer, and the answer is no. The rest have published nothing about reading yours, in either direction.

Google says it outright. The mythbusting section of its generative AI guide reads:

LLMS.txt files and other "special" markup: You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them.

The recap repeats it. Google tells site owners they can ignore tactics like creating unnecessary AI text files. That covers AI Overviews and AI Mode, which are generative features built on Search rather than separate products. It sits alongside what Google documents about that tab.

The others document robots.txt instead. OpenAI’s crawler documentation lists four user agents with different purposes. OAI-SearchBot and GPTBot use robots.txt directives for search inclusion and model training respectively. OAI-AdsBot checks pages submitted as ChatGPT ads. ChatGPT-User handles user-triggered actions, and OpenAI states that robots.txt rules may not apply to those requests.

Anthropic's guidance for site owners names ClaudeBot, Claude-User, and Claude-SearchBot, and says opting out requires editing robots.txt. Perplexity documents two agents with different behavior. PerplexityBot follows robots.txt directives and uses published IP ranges. Perplexity-User handles user-triggered requests and generally ignores robots.txt. None of the three providers documents support for llms.txt.

Publishing a file is not reading yours. llmstxt.org notes that OpenAI, Anthropic, and Gemini publish llms.txt files for their own developer docs. Perplexity's documentation site serves one too. Those are claims about what they serve, not about what their engines fetch.

The conclusion should change if an engine publishes documentation stating that it fetches and uses the file, just as providers document their supported crawlers. Until that happens, a vendor telling you llms.txt drives AI visibility is describing a hope. Ask which page the claim comes from, and check whether that page belongs to the engine or to the spec.

How llms.txt Differs From robots.txt and a Sitemap

An llms.txt file grants nothing and blocks nothing. robots.txt tells automated tools what access is acceptable. A sitemap lists the indexable pages a search engine should know about.

The spec expects all three to coexist. Sitemaps list the URLs you want search engines to discover. llms.txt offers a curated overview instead, and can complement robots.txt by giving context for allowed content. Structured data stays a separate job, with its own documentation, and the file can point at it.

Writing an llms.txt File, With an Example

The format has one required element and several optional ones. In specification order, it allows an optional byte-order mark, requires an H1 containing the site name, and may include a blockquote summary, non-heading content, and H2 sections containing lists of Markdown links. Only the H1 is required.

Here is an llms.txt example built to that structure:

# Northwind Supply

> Northwind Supply sells industrial fasteners to UK manufacturers. This file lists the pages

> an agent needs to answer questions about products, lead times, and returns.

Prices on product pages exclude VAT. Lead times assume UK mainland delivery.

## Products

- Fastener catalogue: Full product list with specifications and stock codes

- Materials guide: Grades, coatings, and corrosion ratings

## Policies

- Returns and warranty: Windows, conditions, and the RMA process

## Optional

- Company history

That file is constructed to the spec, not lifted from a live site. For a published one, the specification points at the FastHTML documentation file. It sits at /docs/, so it covers the documentation pages and nothing else.

Whether to Write It or Generate It

An llms.txt generator already ships with several platforms. llmstxt.org lists Mintlify, GitBook, and Wix as generating one automatically. It names Yoast SEO and AIOSEO as WordPress plugins that do the same. Read that as publishers and tooling adopting the format, not as engines consuming it.

Hand-writing wins on one axis. A generated file usually reflects the site’s existing structure. A curated file lets you prioritize the pages that matter most. That distinction becomes useful on a documentation site with hundreds of pages.

The specification suggests a simple test: give an agent your llms.txt file as its starting point, then ask questions about your content. If it cannot answer, your links or your descriptions are the problem. Write descriptions, not labels, and drop jargon you have not explained anywhere the file points to.

llms-full.txt and When It Is Worth Generating

llms-full.txt is not defined in the v2 specification. It is a platform convention that developed alongside the proposal.

It combines the content of an entire documentation site into one file. Mintlify's documentation describes llms-full.txt as combining an entire documentation site into one file. Each page contributes its title, source URL, description, and full markdown content. Mintlify hosts one automatically at the project root.

Generate it when your platform does it for free. The readers who benefit are coding agents pulling a library's docs into context. Skip the hand-maintained version. A manually maintained version rarely makes sense for a marketing site because it duplicates existing content and becomes outdated quickly.

What llms.txt Can and Cannot Do

No documented search benefit. Google explicitly states that llms.txt neither helps nor harms visibility in Google Search. The other providers reviewed here have not published evidence that they use it for ranking or retrieval.

Lighthouse flags it, and that is less than it sounds. Chrome's agentic browsing audit reports a server error when it cannot fetch the file. A missing file is marked Not Applicable rather than failed, because providing one is optional.

That makes it a cost question rather than a strategy question. The practical recommendation for most websites is narrow. Take the generated file if your CMS makes one. Skip the hand-maintained version unless you run a documentation site. Expect no ranking effect from either.

Coding agents are the strongest documented use case. The v2 changes log says they use these files reliably, which helps explain why documentation platforms generate them automatically. An API is a real reason to publish one, and it has nothing to do with search visibility.

What a model says about your brand can draw on both your own site and the off-site signals that shape an answer. On-site technical work improves access and clarity, while third-party sources provide additional corroboration.

Because you control the file, it cannot provide the independent corroboration that third-party sources can. It may make your own information easier to navigate, but it does not overcome the limits of publishing only on your own domain.

Where llms.txt Fits in a Klarivo Program

On-site hygiene is a small part of a visibility program. llms.txt is a small part of on-site hygiene. Klarivo treats it the way the published evidence supports: ship the free version, then spend the afternoon on work an engine has documented.

The question worth answering is which sources an engine reads when a buyer asks about your category. Klarivo Monitor tracks brand visibility across ChatGPT, Claude, Perplexity, Gemini, and Grok. It reports the domains those answers drew on. That list shows which domains shaped the answers and often includes third-party pages where additional work may be needed.

Klarivo's Reddit and community work exists for the same reason. Many conversations that shape buyer decisions sit on domains a brand does not own. Building a credible presence in those communities can strengthen the third-party footprint engines encounter.

Want to know which AI-search changes are worth implementing and which can wait? Book a 15-minute Klarivo discovery call.

Frequently Asked Questions About llms.txt

Where should the file live on a large site?

At the site root, or at any subpath covering a coherent set of pages. The spec says a file covers the URLs under its path, and that agents should use the most specific one. A docs-only file at /docs/llms.txt is valid, and often better than one giant root file.

Should I list every page in it?

No. The file is meant to stay small enough to fit in context, with the detail behind the links. A generated index that lists everything works as a default. A curated one that identifies your ten most important pages is closer to the intent.

Do I still need robots.txt if I publish llms.txt?

Yes, and robots.txt is the more consequential of the two. OpenAI's opt-outs for training and for search both run through robots.txt directives. Nothing in the llms.txt proposal replaces them.

How can I tell whether an agent read my file?

Check server logs for requests to that path, grouped by user agent. No engine reports it back to you. You get a fetch count, and no way to connect it to an answer anyone saw.

Does a stale file do harm?

Nobody has published evidence either way. The plausible risk is an agent following links to pages you deleted, which argues for generating the file rather than maintaining one by hand.