llms.txt: The Reality Check

llms.txt is a proposed convention — a plain-text or markdown file at your site root that offers an LLM-friendly index of your content. It was suggested by Jeremy Howard of Answer.AI in September 2024, but as of now no major AI engine consumes it for live retrieval, and its measured effect on AI citations is marginal to unmeasurable for most sites. Treat it as low-cost insurance and a better experience for developer audiences — not as a ranking lever.

Where llms.txt came from

Jeremy Howard (Answer.AI) proposed llms.txt in September 2024. The idea borrows from robots.txt and sitemap.xml: a predictable plain-text or markdown file that sits at your site root and is easy for machines to find.

The original proposal describes two files:

The intent is to give language models a clean, curated map of what matters on your site, rather than leaving them to parse cluttered HTML.

The adoption reality

Here's the honest picture. No major AI engine consumes llms.txt for live retrieval. Google states it directly: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in generative AI search." OpenAI, Anthropic, and Perplexity have never committed to reading the file either.

The server-log evidence backs this up. A 90-day study by Otterly in 2026 logged more than 62,100 AI-bot visits — of those, just 84 requests (0.1%) touched /llms.txt. That's roughly three times less traffic than an average content page receives, and the study found no correlation between having an llms.txt file and AI crawler activity.

So the empirical effect on AI citations is marginal to unmeasurable for most sites. Don't sell it as a ranking lever. If you want to understand what actually drives visibility in AI search, AEO & GEO and AI Search Engines are better places to spend your energy.

Who actually benefits

Adoption is concentrated in developer-tool documentation sites — Anthropic, Mintlify, Cloudflare, Vercel, Stripe, Supabase, and LangChain among them. The reason is practical, not algorithmic: developers routinely paste documentation URLs into Claude or ChatGPT, and a clean llms.txt index improves that manual workflow.

If your audience includes developers who work alongside AI assistants, the file earns its keep by making your docs easier to feed into a model on demand. If your audience is general consumers, the payoff is much thinner. Either way, treat llms.txt as low-cost insurance and better UX for a specific audience — not as a promise about how you'll appear in AI answers.

What to put in /llms.txt

Keep the structure simple: a title, a one-line description, some context, and curated sections of links with short descriptions.

# Site Name

> One-sentence description of what this site/product is.

Key context: ideal customer, what makes this canonical, licensing.

## Docs {#docs}
- [Getting Started](https://example.com/docs/start.md): 5-min quickstart
- [API Reference](https://example.com/docs/api.md): full REST API

## Guides {#guides}
- [Authentication](https://example.com/guides/auth.md): OAuth2 + API keys

## Optional {#optional}
- [Changelog](https://example.com/changelog.md)

A few conventions have emerged among the sites that use it:

• Provide a .md version of every important HTML page at the same path. For example, /docs/auth gets a companion /docs/auth.md. Mintlify and Anthropic both do this.
• Keep llms.txt under about 50 entries. Use the ## Optional section for lower-priority links that a model can skip when its context window is constrained.
• Make llms-full.txt the entire concatenated docs corpus, ideally under 500K tokens.

Build-system recommendations

Don't hand-maintain these files. A hand-edited index drifts out of date the moment your docs change, which defeats the purpose.

• Auto-generate the file from your CMS or docs build so it stays in sync with your content.
• Serve it with the right header: Content-Type: text/markdown; charset=utf-8.
• Signal the markdown sibling in HTML. Include <link rel="alternate" type="text/markdown" href="/path.md"> in the head of each HTML page that has a markdown version.

These steps are the difference between a file that quietly reflects your live content and one that becomes stale technical debt.

How this fits your wider strategy

llms.txt is closely related to how you manage crawler access generally. If you're deciding what AI bots can and can't reach, robots.txt for AI Bots is the more consequential file, since it governs actual crawler behavior rather than offering an optional index.

For the broader shift toward assistant-driven browsing that makes conventions like llms.txt plausible in the first place, see The Agentic Web and The Zero-Click, Agent Future. And if you're auditing where to invest limited time, Dead SEO Tactics in 2026 helps you separate speculative conventions from proven work.

What to do

1. Decide whether your audience includes developers who paste your docs into AI assistants. If not, deprioritize llms.txt entirely.
2. If you proceed, set expectations honestly with stakeholders: this is insurance and developer UX, not a citation or ranking lever.
3. Auto-generate /llms.txt from your CMS or docs build so it never drifts out of date.
4. Keep the index under ~50 entries and push lower-priority links into the ## Optional section.
5. Publish a .md version of each important page at the same path, and add the <link rel="alternate" type="text/markdown"> tag to those HTML pages.
6. If you maintain a large docs corpus, generate /llms-full.txt as the concatenated content, aiming to stay under 500K tokens.
7. Serve every file with Content-Type: text/markdown; charset=utf-8.
8. Spend the bulk of your visibility effort elsewhere — on content quality, entities, and the fundamentals that AI engines actually rely on today.