Markdown for agents: how to serve your pages to AI agents—and what it won’t fix

Most pages are roughly 30 times heavier than the text a human reads, and an AI agent pays for every wasted token. Serving Markdown to those agents has become an afternoon’s work with three proven delivery paths—and it still won’t fix your weak content, your blocked crawler, or your missing schema.

The short answer

Agents read your site the way any HTTP client does: they fetch a URL and parse whatever comes back. Making your site readable for ChatGPT, Perplexity, or Claude means sending them the content without the wrapper. Three delivery paths are in production today:

• Content negotiation: when a request’s Accept header lists text/markdown, your server answers the same URL with a Markdown representation, labeled Content-Type: text/markdown; charset=utf-8, with a Vary: Accept header so caches keep the two representations apart.
• Static .md mirrors: publish a Markdown copy at a second URL—the same path with .md appended is the common convention—and declare it with so agents can discover it. The llms.txt spec recommends exactly this pattern.
• Edge conversion: Cloudflare’s Markdown for Agents converts HTML to Markdown at the CDN edge for enabled zones, triggered by that same Accept header. No origin changes, no second URL.

None of these require a second website or a rewrite. The right choice depends on your stack, your Cloudflare plan tier, and how much of the machinery you want to own.

Why HTML is expensive for agents

Agents don’t read pixels; they read tokens, and context is a budget. A typical page ships the same navigation, footer, scripts, and theme markup on every URL, wrapped around a few hundred words of actual content. One independent benchmark of 193 pages from 66 domains put the median markup-to-content ratio at 30.3x—the HTML shell is 30 times the size of the words it carries. Even a plain conversion isn’t free of structure: the same study found the text-only output still runs 2.8 times the visible text in tokens, because link syntax and layout carry token cost of their own.

“This blog post you’re reading takes 16,180 tokens in HTML and 3,150 tokens when converted to markdown. That’s an 80% reduction in token usage.” — Cloudflare, measuring its own announcement post

That’s Cloudflare’s measurement from February 2026, and independent numbers land in the same neighborhood, sometimes better. The 193-page study found plain HTML-to-Markdown conversion cuts 90.9% of tokens, and main-content extraction cuts 98.1%. A separate 20-page measurement reported a median raw HTML page of roughly 128,000 tokens against 3,241 for extracted main-content Markdown—but with a savings range of 18% to 90%, which is a warning against quoting any single blended figure.

Here’s why this matters for whether an agent cites you. Context windows are finite and fetch budgets are stingier still. A page that arrives whole in 3,000 tokens gets read to the end; the same page at 128,000 tokens gets truncated, summarized on the fly, or never fetched in full at all. Token efficiency isn’t a cost optimization for agents—it’s a readability optimization.

Two honest caveats from the 193-page study. First, Markdown link syntax is part of the remaining cost—that’s why converted output still exceeds the visible text. Second, aggressive main-content extraction silently deleted about 19% of the text that existed only on that page: footnotes, tables, asides. If you extract, extract carefully.

How the three paths compare

Same content, three very different ownership models. Here’s how they trade off for a small-business site.

Path 1: Content negotiation

Agents send an Accept header on every fetch, and some—OpenCode among them—already rank text/markdown first. Proactive content negotiation, answering the same URL with a different representation, is standard HTTP. Mintlify runs it in production for its docs platform and reports a 30x efficiency gain over serving raw HTML. Freshness is automatic because there’s one pipeline and one URL—the Markdown is born from the same content as the HTML, the moment it changes. The cost: you own the conversion in your application. Mintlify also sends X-Robots-Tag: noindex, nofollow on its Markdown responses so search engines don’t index the agent variant—steal that detail if you build this.

Path 2: Static .md mirrors

Build-time mirrors are the cheapest path that works on any host, including Cloudflare’s free tier: your build renders each page twice, once as HTML and once as Markdown, and you serve the Markdown at /blog/post.md. Vercel documents this pattern for documentation sites, down to a rewrite destination of ${prefix}/:path*.md. The failure mode is freshness. A mirror is a cache that goes stale the day someone edits the CMS without rebuilding. If your content changes weekly, automate the mirror or pick a different path.

Path 3: Edge conversion

Cloudflare’s Markdown for Agents converts at the edge: no build step, no template work, one dashboard toggle. The trade-offs are plan tiers and hard limits, which is what the next section is about.

Cloudflare Markdown for Agents, in detail

Announced February 12, 2026, Markdown for Agents converts HTML pages requested from Cloudflare’s network into Markdown in real time—for enabled zones, when the client’s Accept header includes text/markdown. The output strips headers, footers, navigation, scripts, and styles, keeps your JSON-LD structured data as code blocks, and adds YAML frontmatter at the top.

Availability: it’s in beta at no cost for Pro, Business, and Enterprise plans, plus SSL for SaaS customers. It is not on the free plan—free-tier sites have to upgrade or use one of the other two paths.

Verification is one line of curl, straight from the feature docs. You should get Markdown back, not HTML:

curl https://blog.cloudflare.com/markdown-for-agents/ -H "Accept: text/markdown"

To inspect the headers instead of the body, swap in curl -sI and confirm the content type before you trust what you got back—a 200 only proves the server answered.

The limits are documented and worth respecting: Cloudflare only converts from HTML, and conversion is capped at 6 MiB of decompressed HTML per page. The HTML-only part deserves emphasis. If your page’s content exists only after JavaScript runs—the server ships an empty shell and a framework fills it in—the edge sees the shell, and the Markdown reflects the shell, not your content. That isn’t Cloudflare being sneaky; it’s the same reason crawlers have always struggled with JS-first pages. Check what your origin actually sends before assuming the conversion is doing anything.

Build it into your own stack

Our stack is PHP with Twig templates, so I’ll describe the wiring I’d use there—but the pattern is identical in any server-rendered framework, and the same logic applies to WordPress with a custom plugin.

The cheapest correct version isn’t a converter at all; it’s template discipline. If your main content lives in one Twig block and everything else—nav, footer, sidebar—lives in the layout, you can render a second template that extends a bare layout and prints only that block. Your CMS content is already structured; you’re reusing that structure instead of scraping it. Detect the Accept header before rendering, and branch:

if (str_contains($_SERVER['HTTP_ACCEPT'] ?? '', 'text/markdown')) {
    header('Content-Type: text/markdown; charset=utf-8');
    header('Vary: Accept');
    header('X-Robots-Tag: noindex, nofollow');
    echo $twig->render('post/markdown.twig', $context);
    exit;
}
echo $twig->render('post/html.twig', $context);

Three details separate a toy from a production path:

• Send Vary: Accept so any cache between you and the agent stores the HTML and Markdown representations separately. Without it, a cache can serve Markdown to humans or HTML to agents.
• Keep URLs and links intact. Relative links in the Markdown should resolve against the page URL, and a canonical URL in the frontmatter keeps the agent anchored to the HTML original.
• Declare the Markdown variant on the HTML page with . The llms.txt spec recommends standard link relations for exactly this discovery.

Honest disclosure: our own site ships a llms.txt but no Markdown surface yet. This section is the plan we’re working through ourselves, not a war story about one we already run.

What Markdown delivery won’t fix

Markdown makes your existing content cheaper to read. It does not make that content better, more findable, or more citable. Several things still gate everything upstream or downstream of delivery:

• Crawl access gates all of it. If your robots policy, your CDN’s bot settings, or a tool like Cloudflare’s AI Crawl Control blocks the agent, it never sends the Accept header in the first place. Delivery fixes nothing for a client that can’t reach you.
• Weak content stays weak. A thin page converted to Markdown is a thin page in fewer tokens. Agents cite pages because the content answers a question, not because it was served efficiently.
• Schema and entity clarity still carry the citations. Structured data is how an agent connects your page to the thing it’s about. Cloudflare’s converter preserves JSON-LD precisely because the format still matters once the wrapper is gone.
• Content negotiation is a convention, not a contract. Per the HTTP standard, a server that ignores the Accept preference and returns HTML anyway is fully conformant—agents must tolerate being served HTML, and some will fall back to it silently.
• Bad extraction is worse than no conversion. A pipeline that deletes 19% of the unique text on a page to save tokens has traded substance for speed.

Verify it in five minutes

None of this is real until you’ve watched it happen. Three read-only checks, five minutes:

1. Fetch as an agent: curl -s https://example.com/your-key-page -H "Accept: text/markdown" | head -40. You should see clean Markdown, not HTML. If you get HTML, the path isn’t live for that URL.
2. Check the headers, not just the body: curl -sI https://example.com/your-key-page -H "Accept: text/markdown". Confirm Content-Type is text/markdown and, if you built the path yourself, that Vary: Accept is present.
3. Watch what agents actually send. Claude Code’s header is Accept: text/markdown, text/html, */*—Markdown listed, but with equal weight, so a tie-breaking server will serve HTML. OpenCode is more explicit: text/markdown;q=1.0, text/x-markdown;q=0.9, text/plain;q=0.8. Log incoming Accept headers on your origin for a week and you’ll know who’s asking instead of guessing.

For the Cloudflare path, the feature’s docs verify with the same one-liner against Cloudflare’s own blog post—so you can test their zone first and yours second. If theirs converts and yours doesn’t, your zone isn’t enabled.

Token efficiency as a citation factor

This is the part I’d defend hardest against skeptics who call agent-friendly formatting a fad. Citability is a function of being read completely. An agent that gets your whole argument in 2,500 tokens can quote it, attribute it, and link it; the same argument arriving inside 60,000 tokens of markup fights truncation the entire way down. Everything agents reward—fresh content, clear structure, unambiguous entities—works better when the delivery layer stops charging them for your navigation.

It also compounds with the other work. llms.txt, proposed by Jeremy Howard at Answer.AI on September 3, 2024, was framed around exactly this problem: constructing the right context for an LLM from a website was ambiguous. Markdown delivery is the per-page layer underneath the site-wide map llms.txt provides—one tells the agent where to look, the other makes looking cheap.

What to do this week

Work down this list and stop at the first row that describes you:

1. You’re on Cloudflare Pro, Business, or Enterprise: enable Markdown for Agents in the dashboard, then verify with the one-line curl. Budget 15 minutes, most of it finding the toggle.
2. You’re on Cloudflare free, or not on Cloudflare: choose between .md mirrors for your top pages (build-time, works anywhere) and full content negotiation (one-time app work, best long-term).
3. Your pages rely on JavaScript to render content: fix what the server sends before adding any Markdown path. There’s nothing to convert until plain HTML carries your content.
4. Your pages are already lean, text-first, and small: doing nothing is a legitimate answer. Add Accept-header logging, watch for a month, and revisit if agent traffic shows up.

The whole decision fits in half an hour because the paths are complementary, not competing—you can enable edge conversion today and negotiate properly later.

When it’s worth getting help

Here’s where this sits in the honest priority order we use in our AI Visibility Audits: access first, schema second, Markdown delivery third. A site that blocks AI crawlers has nothing to deliver. A site agents can reach but can’t interpret—no structured data, unclear entities—will not be rescued by a leaner wrapper. Markdown delivery is the optimization you add once the two layers beneath it are solid, and in audits like this it’s usually the quickest win of the three, precisely because it changes nothing about your content.

If you want that ranking worked out for your own site, the audit walks all three layers and ends with a fixed to-do list. And because delivery changes only matter if citations move, we track how ChatGPT, Claude, Perplexity, and Gemini actually describe a brand over time—that’s what we built Cited for, and it’s how you’d know the Markdown work paid off.