llms.txt AI search SMB technical SEO AEO

llms.txt: the spec-compliant format guide for SMBs (step-by-step)

How to build an llms.txt file AI crawlers can actually navigate—and why a file full of marketing prose is worse than not having one at all.

Alexander Kirsch-Clayton By Alexander Kirsch-Clayton, Founder ·10 min read

A spec-compliant llms.txt is a curated map of your site written for language models: an H1, a one-line summary, and sections of Markdown links. Most files I audit aren’t that. They’re paragraphs of marketing copy with no URLs—and that version is worse than having no file at all.

Why AI crawlers need a navigation map

GPTBot, OAI-SearchBot, and PerplexityBot arrive at your site with no inherent sense of what matters. They see HTML wrapped in navigation, ads, and JavaScript, and they have to guess which pages are worth citing. As Search Engine Land put it, “converting complex HTML pages with navigation, ads, and JavaScript into LLM-friendly plain text is both difficult and imprecise.”

Here’s the cleanest way to think about the three files that live at your site root. robots.txt tells crawlers what’s forbidden. An XML sitemap tells them what exists. llms.txt tells them what’s important. It’s the relevance layer, and until recently nobody was writing one.

The traffic makes the case on its own. Cloudflare’s 2025 crawler study found GPTBot’s share of requests rose from 2.2% to 7.7%—a 305% jump in a year. PerplexityBot, starting from a near-zero base, posted by far the steepest growth rate in the same study. These bots are showing up. The question is whether they find a curated entry point or a wall of HTML.

What is llms.txt—and who specified it?

The /llms.txt proposal came from Jeremy Howard, co-founder of Answer.AI and fast.ai, published on September 3, 2024. His reasoning was simple: constructing the right context for an LLM from a raw website is ambiguous, so let the people who know the site best do the curating. “Site authors know best,” Howard wrote, “and can provide a list of content that an LLM should use.”

This is not an official standard from Google or OpenAI. It’s a community convention that the major AI crawlers can read because it’s plain Markdown. The spec chose Markdown deliberately: “at the moment the most widely and easily understood format for language models is Markdown,” because “we expect many of these files to be read by language models and agents.” No XML, no schema wrapper—just text a model already parses fluently.

I want to be honest about the evidence, because the hype has outrun it. An SE Ranking study across roughly 300,000 domains found no relationship between having an llms.txt and how often a domain gets cited—“at least not yet.” Google’s John Mueller has said flatly that no AI system currently uses llms.txt. And an Ahrefs analysis of server logs from 137,000 domains found that 97% of deployed files received zero AI requests in May 2026, even as deployments grew 8.8× in a year.

Anatomy of a spec-compliant llms.txt

The official spec at llmstxt.org defines an exact, ordered sequence. Only one element is strictly required, but the full pattern is what makes the file useful:

  1. An H1 with the name of your site (# Acme Bakery). This is the only required section.
  2. A blockquote summary—one or two sentences with the key context a model needs to interpret the rest (> Family-run bakery in Leipzig, since 2009.).
  3. Optional prose sections (no headings) with detail on how to read the file.
  4. One or more H2 sections holding “file lists”—URLs where further detail lives.

Each entry in a file list takes the form [name](url), optionally followed by : a one-line note. That note is the whole point—it’s how you tell the model why the page matters. Here’s a minimal, spec-compliant example:

# Acme Bakery

> Family-run bakery in Leipzig. Wholesale and retail, baking since 2009.

## Services

- [Wholesale ordering](https://acme.example/wholesale): Standing weekly orders for cafés and restaurants.
- [Custom cakes](https://acme.example/cakes): Made-to-order celebration cakes, 72h lead time.

## About

- [Our story](https://acme.example/about): Who we are, where we bake, and how to reach us.

There’s one more structural detail worth knowing. The spec reserves a special ## Optional section: “if it’s included, the URLs provided there can be skipped if a shorter context is needed.” In practice that means only your genuinely core pages should sit outside ## Optional. Everything secondary—comparison pages, niche landing pages—goes inside it, so a context-constrained crawler drops the right things first.

Before / after: what goes wrong with a prose llms.txt

The most common broken file I see isn’t malformed Markdown. It’s prose. Someone pasted their homepage intro into a text file and shipped it. It looks like this:

# Welcome to Acme Bakery!

We are a passionate, family-owned bakery dedicated to crafting the
finest artisanal breads and pastries using time-honored techniques
and only the freshest locally sourced ingredients. Our mission is to
delight every customer, every single day. Visit us and taste the
difference real craftsmanship makes!

Read it as a crawler would. There are no URLs to follow, no sections to prioritize, no facts to anchor to. Three things are going wrong at once:

  • It spends crawl budget with nothing to show for it. A bot fetches the file, parses it, and comes away with no navigable structure—no next step.
  • It invites fact conflicts. Marketing prose tends to carry stale or fuzzy claims—an old location, a service you dropped, a founding year that doesn’t match your other pages. Models cross-check against the rest of the web, and a contradiction reads as a trust problem.
  • It gives the model no machine-readable page list. Without one there’s no navigable context, which is the entire job the file exists to do.

This is the honest framing, and I’ll flag it plainly: I’m not aware of a controlled study measuring prose files against structured ones. The claim below is reasoned from the spec’s design intent, not an A/B result. But the logic is hard to argue with.

An llms.txt file written as prose is worse than having none at all: it spends an AI bot’s crawl budget without providing a navigable structure—the equivalent of an XML sitemap that contains only marketing copy.

Step-by-step: build your llms.txt in under 45 minutes

You don’t need a tool for this. You need your sitemap open in one tab and a text editor in the other. Five phases:

1. Audit your pages

List every canonical URL on the site. Not variants, not tag archives—the real, indexable pages. Ten minutes for most small business sites.

2. Set priority

For each URL, ask one question: do I want an AI model to cite this? Your services, your about page, your contact page, your best blog posts—yes. Your privacy policy and your tag pages—no, or into ## Optional.

3. Name your sections

Group the survivors into H2 sections. For an SMB, ## Services, ## About, ## Blog, and ## Contact cover almost everything. Push secondary pages into ## Optional.

4. Write one line per URL

This is where the file earns its keep. Each description is one benefit statement—what the reader gets on that page—not a buzzword salad. “Flat-rate audit of how ChatGPT and Perplexity describe your brand” beats “Cutting-edge, synergistic AI solutions.” Keep it to a line.

5. Deploy at the root

The file lives at https://yourdomain.com/llms.txt—site root, same neighborhood as robots.txt. Serve it as UTF-8 plain text. That’s it; the root path is the convention crawlers look for. (Some teams also add a <link> reference in the head, but that’s community practice, not part of the spec—don’t let it block you.)

The five most common llms.txt mistakes

The llms.txt Generator project names the top formatting failure bluntly: “dumping every URL with no descriptions (it’s a map, not a sitemap).” That’s mistake one, and it’s the majority of broken files. Here’s the full set I keep running into:

  1. Prose instead of a URL index. Background copy with no links to follow. Covered above—it’s the cardinal sin.
  2. Factual inconsistencies. Location, founding year, pricing that contradicts your other pages. Models cross-check these against other sources and treat a mismatch as a signal conflict.
  3. Omitting the blog or resources section entirely. Your best citation candidates are often long-form posts. Leave them out and you’ve hidden your most quotable content.
  4. Duplicate or generic one-liners. “Learn more about our services” ten times over tells a model nothing. Each line should distinguish its page.
  5. No pointer to llms-full.txt on content-heavy sites. If you maintain a companion file (next section), reference it so crawlers that want depth can find it.

In the audits I run, mistake two—contradictory location details—is the fastest trust-killer to fix. A file that says one city while your footer, your Google profile, and your legal page say another is a five-minute correction that removes a real conflict signal. Fix it first.

Advanced: llms-full.txt for content-heavy sites

Think of llms.txt as a table of contents and llms-full.txt as the whole book. As GitBook describes it, the standard file “is usually a guide to your best sources,” while llms-full.txt “contains a fuller export of your documentation in one file.” Instead of linking out to pages, it inlines the actual content.

That’s overkill for a typical five-page marketing site. It earns its place on documentation sites, API references, knowledge bases, and news portals—roughly 50 pages and up, where letting a model ingest everything in one fetch beats making it crawl link by link. You maintain both: llms.txt for navigation, llms-full.txt for depth, with the index file pointing at the full one. One naming note: llms-full.txt is a widely used community convention, not verbatim in the original proposal—so don’t expect every crawler to request it. Documentation-focused agents are the main consumers today.

Validation: testing your llms.txt for spec compliance

Three checks, fastest to most meaningful.

a) Headers, via curl

Confirm the file is reachable and served as text:

curl -I https://yourdomain.com/llms.txt

You want a 200 status and a content type of text/plain; charset=utf-8 (or text/markdown). A 404, an HTML error page, or a wrong charset means crawlers won’t parse it cleanly.

b) Manual pass against the spec checklist

Walk the file against the six-point checklist from the build section. One H1, a summary blockquote, H2-grouped links, a real one-liner on every entry, secondary pages in ## Optional. No broken Markdown link syntax—that’s the second-most-common failure after sitemap dumps.

c) Ask a model directly

The real test is behavioral. Before you deploy, ask ChatGPT or Perplexity: “What are the main services of [yourdomain]?” Note the answer. Deploy, wait for a recrawl, and ask again. If the description gets sharper and more accurate, the file is doing its job. Because recrawl timing is unpredictable and citations shift, this is worth tracking over weeks rather than eyeballing once—a monitor like Cited watches how ChatGPT, Claude, Perplexity, and Google AI Overviews describe your brand over time, which turns “I think it helped” into an actual before-and-after. If you’d rather have someone run that baseline for you, that’s exactly what our AI Visibility Audit does.

Live example: Digital Domination’s revised /llms.txt

Here’s our own file, trimmed to a template you can adapt in one sitting. Note the blockquote summary, the benefit-first one-liners, and the secondary pages parked in ## Optional per the spec.

# Digital Domination

> AI-search visibility and managed WordPress for small businesses.
> Based in Minnesota, working with clients across the EU and US.

## Services

- [AI Visibility Audit](https://digitaldomination.xyz/en/services/ai-visibility-audit): Flat-rate audit of how ChatGPT, Perplexity, and Google AI Overviews describe your brand. Starting at 99 €.
- [Managed WordPress hosting](https://digitaldomination.xyz/en/wordpress-hosting): Fast hosting with updates and backups handled for you.
- [WordPress migration](https://digitaldomination.xyz/en/wordpress-migration): Move an existing site onto our stack with no downtime.
- [Website build](https://digitaldomination.xyz/en/website-build): New marketing sites built for AI and human readers alike.
- [Private AI & agents](https://digitaldomination.xyz/en/ai): Self-hosted AI and custom agents for teams that can’t send data to third parties.

## About

- [Services overview](https://digitaldomination.xyz/en/services): The full list of what we do and who it’s for.
- [Contact](https://digitaldomination.xyz/en/contact): Request a quote or book a call.

## Blog

- [Blog index](https://digitaldomination.xyz/en/blog): Guides on AI search visibility, llms.txt, and WordPress.

## Optional

- [WordPress hosting comparison](https://digitaldomination.xyz/en/wordpress-hosting-comparison): How our hosting compares to common alternatives.
- [For vegan brands](https://digitaldomination.xyz/en/vegan): Visibility work specifically for vegan businesses.

Last thing, and it’s the part most people skip: this file is not a one-time task. It drifts the moment you publish a post you’d want cited or change a price. Build a five-minute update into the same routine—ship a blog post, add its line to ## Blog; change a rate, fix the number here too. The whole value of llms.txt is that it’s the one place you’ve promised a model the truth. Keep that promise current.