llms.txt Explained: What It Is, Why You Need It, and How to Create One

Your site has a robots.txt. It tells Google what to crawl. It’s been sitting in your root directory for 20 years, quietly doing its job.

Now there’s a new file doing a similar job for a completely different audience: AI models. It’s called llms.txt, and if you haven’t heard of it yet, you’re about to be very glad you opened this tab.

Here’s the short version: llms.txt is a plain-text file you place at the root of your website that tells large language models how to understand your site. Not crawl it. Understand it. And in a world where ChatGPT, Claude, and Perplexity are increasingly the front door to the internet, that distinction matters a lot.

Let’s get into it.

The problem: AI models are terrible at reading your website

Here’s a thing nobody wants to admit. When ChatGPT or Perplexity pulls your site into a response, it’s not reading it like a human does. It’s grabbing a chunk of HTML, stripping out the navigation, headers, scripts, ads, cookie banners, and footer — and hoping whatever’s left is the actual content.

Most of the time? It’s wrong. Or it gets confused. Or it cites the wrong page. Or it just skips your site entirely because parsing it is too expensive.

This is the same problem Google faced in 1994, which is why robots.txt was invented. But robots.txt solves the wrong problem for AI. It controls access — what a crawler can and can’t fetch. It doesn’t help a model understand what your site is about, where the important content lives, or how to summarize it accurately.

That’s the gap llms.txt is trying to fill.

What llms.txt actually is

llms.txt is a markdown file placed at https://yoursite.com/llms.txt. It’s a structured, human-and-machine-readable summary of your site, designed to be consumed by language models during inference.

Think of it as a curated table of contents written specifically for an AI that has maybe 10 seconds to figure out what your company does.

The spec was proposed by Jeremy Howard (the guy behind fast.ai and Answer.AI) in late 2024. Within months, it was picked up by Anthropic, Cursor, Vercel, Stripe, Hugging Face, and hundreds of developer-first companies. As of April 2026, the list of sites publishing an llms.txt has crossed 15,000 domains and is doubling every quarter.

It’s not an official standard yet. There’s no W3C stamp on it. But adoption is happening organically, which is usually the better signal anyway.

Why you need one (the cost of not having it)

If you’re not on llms.txt, here’s what’s happening when an AI tries to use your site:

It scrapes your HTML, gets confused by your React hydration, and gives up.
It reads your homepage and incorrectly guesses what you do.
It cites a three-year-old blog post instead of your current product page.
It hallucinates your pricing.
Or — most commonly — it ignores you entirely and cites a competitor who made themselves easier to understand.

We’ve seen this play out across every audit we run. Sites with solid traditional SEO but zero AI visibility. Their content is there. Google loves it. But when you ask Perplexity “what’s the best X for Y,” they don’t show up. Meanwhile a smaller competitor with half the backlinks but a tight llms.txt gets cited in every other answer.

Recent data from the Ahrefs AI Search Study (Q1 2026) found that sites with a published llms.txt were cited 3.2x more often in AI Overviews and Perplexity responses than matched peers without one. That’s not a rounding error. That’s the difference between being discovered and being invisible.

Want to see if your site is invisible to AI search? Run a free audit — it checks for llms.txt, schema, and citation readiness in under 60 seconds.

How to create an llms.txt file

The spec is intentionally simple. It’s markdown. You already know markdown. Here’s the structure:

# Your Company Name

> One-sentence description of what your company does.

Short paragraph giving context. What problem you solve, who you serve, what makes you different. Keep it under 150 words.

## Core Pages

- [Homepage](https://yoursite.com): Brief description
- [Product](https://yoursite.com/product): What this page covers
- [Pricing](https://yoursite.com/pricing): Plans and cost

## Documentation

- [Getting Started](https://yoursite.com/docs/start): Quickstart guide
- [API Reference](https://yoursite.com/docs/api): Full API docs

## Blog

- [Latest posts](https://yoursite.com/blog): Updates and guides

## Optional

- [Changelog](https://yoursite.com/changelog): Product updates

That’s it. Headings, bullet links, short descriptions. The model reads it top to bottom and uses it as a map.

A few rules from the spec:

The file MUST live at the root: /llms.txt
The first H1 is your site name
The blockquote right after is your elevator pitch
H2 sections group related content
Use Optional as a final section for nice-to-haves that the model can skip if context is tight

A real example you can steal

Here’s what a good llms.txt looks like in practice:

# RunAgents

> Autonomous AI agents that run marketing, SEO, and content operations on your behalf.

RunAgents gives you a team of specialized AI agents that handle real marketing work — from running audits to writing content to monitoring brand mentions across AI search. Each agent operates in its own sandbox, follows your brand guidelines, and reports back with actual deliverables. Built for marketing teams who are tired of AI tools that require babysitting.

## Core Pages

- [Homepage](https://runagents.dev): Overview of the platform
- [Free Marketing Audit](https://runagents.dev/free-marketing-audit): Free AI-powered site audit
- [Pricing](https://runagents.dev/pricing): Plans start at $99/mo

## Tools

- [AI Search Optimization](https://runagents.dev/ai-search-optimization): Get cited by ChatGPT and Perplexity
- [LLM SEO](https://runagents.dev/llm-seo): Optimize for large language models
- [Structured Data for AI](https://runagents.dev/structured-data-for-ai): Schema markup guide

## Blog

- [Latest posts](https://runagents.dev/blog): Marketing and AI search guides

## Optional

- [Changelog](https://runagents.dev/changelog): Product updates

Notice what this does well: every link has a description, the pitch is specific (not “we use AI to revolutionize marketing”), and the hierarchy tells the model what matters most.

Common mistakes that kill your llms.txt

After auditing hundreds of these, here are the patterns that show up again and again:

Writing it like marketing copy. Models don’t care that you’re “the leading platform.” They care what you do. Be specific. Use nouns, not adjectives.
Listing every page on your site. This isn’t a sitemap. Curate. 10-30 links is plenty for most sites. If you link to everything, you’re signalling nothing.
Forgetting descriptions. A bare link with no context is almost useless. The model needs the description to decide whether to fetch the page.
Putting it anywhere but the root. It has to be at /llms.txt. Not /public/llms.txt. Not /docs/llms.txt. Root.
Never updating it. If your product changed six months ago and your llms.txt still describes the old version, you’re actively misleading models. Treat it like any other living doc.

How to test if it’s working

Once it’s live, here’s the quick check:

Visit https://yoursite.com/llms.txt in your browser — it should render as plain text, not 404.
Ask ChatGPT or Perplexity to summarize your site. If the description matches your llms.txt, the model has likely pulled it.
Ask a specific question your llms.txt should help with (“what does [your company] do?”). Check if the answer uses language from your file.
Over 2-4 weeks, monitor your brand mentions in AI search. You should see citation frequency go up.

For the more systematic approach, our free marketing audit checks llms.txt presence, structure, and whether it’s actually being used by AI crawlers. Handy if you don’t want to eyeball it.

The bigger picture

llms.txt is not a silver bullet. It’s not going to magically get you cited if your content is bad or your site is invisible for other reasons. But it’s the lowest-effort, highest-leverage thing you can do in an afternoon to make your site readable by the systems that are increasingly deciding what users see.

Two years from now, publishing an llms.txt will be as default as publishing a robots.txt. The people doing it now are getting the early-mover advantage. The people who wait will be catching up.

Fix this in an afternoon

Here’s the move. Spend 30 minutes writing your llms.txt. Deploy it. Check it renders. Move on.

Or — if you’d rather not write it yourself — point our AI Search Optimization agents at your site and they’ll generate one, deploy it, and monitor whether it’s actually improving your citation rate. That’s what they’re built for.

And while you’re thinking about AI visibility, read up on structured data for AI and LLM SEO — they’re the other two pieces of the same puzzle. Get all three right and you stop being invisible to AI search.

Start with the free audit to see where you stand. Then fix what it finds.