An AI agent opens a website’s documentation page to answer a single question. The server returns 90 KB of HTML wrapped around 2 KB of useful text: navigation bars, scripts, cookie banners, footers, tracking code.
The agent discards almost all of it, converts what remains to plain text, and pays for every wasted token along the way. It’s one page—a small loss.
Now multiply that across every request an agent makes, all day, across thousands of sites.
Markdown negotiation reduces that loss. It’s a use of HTTP content negotiation, the standard mechanism the web has carried since the 90s.
When a client requests a page, it can send an Accept header that states which format it prefers. Browsers usually send a longer Accept header that includes text/html plus other acceptable types. AI agents can send Accept: text/markdown.
A server that supports this convention can return clean markdown content, at the same URL, with no separate endpoint or duplicate content needed.
Markdown is the lightweight markup language John Gruber designed for humans to read as plain text. It carries headings, lists, links, and code blocks with almost no surrounding noise.
For AI agents, that means less to parse and a lower token cost.
An HTML response pads content with markup an agent never uses. A single ## About Us heading costs roughly 3 tokens in markdown; its HTML equivalent burns 12 to 15, before you count the wrapper divs, nav bars, and script tags on every page.
Cloudflare measured its own blog post at 16,180 tokens in HTML and 3,150 in markdown, an 80% reduction in token usage.
Read the Docs returns a similar content signal, exposing an x-markdown-tokens header so any client (including coding agents) can read the converted size before it commits a context window.
For publishers, this changes how content reaches an audience. AI agents and AI crawlers now route a growing share of agent traffic that once flowed through search.
An agent that parses your page poorly will summarise it poorly, or skip it. A site that serves clean markdown content gives every coding agent and AI bot a less noisy source, which feeds more accurate citation in the answers those agents generate.
Three implementation routes exist, from no-code to custom:
- Cloudflare converts pages at the edge for enabled zones. This feature is plan dependent, requiring a Pro or Business plan.
- A WordPress plugin can read the Accept header and serve a markdown version for every post and page.
- A developer can write a rewrite rule that detects the header and routes to a markdown endpoint.
The rest of this article covers each in turn.
A short history of a plain-text format
Gruber published markdown in 2004 as a way to write for the web without writing HTML. The goal was a markdown language readable as-is, before any rendering.
A heading is a line that starts with a hash (#). A list is a line that starts with a dash (-). A blank line separates paragraphs, and a line break stays a line break.
That plainness, designed for human writers, turns out to suit machine readers too. Documentation platforms adopted it early, which is why so much technical content already exists in markdown format beneath its rendered HTML.
How HTTP content negotiation works
Content negotiation lets one URL serve more than one representation. The client sends content negotiation headers describing what it wants; the server picks the closest match and returns it. The Accept header field carries that request.
For example, a browser might send Accept: text/html. An agent that wants markdown sends Accept: text/markdown.
Some agents add a quality value to rank their preference, such as text/markdown;q=1.0, text/html;q=0.7, which tells the server to prefer markdown but accept HTML as a fallback.
The server replies with a Content-Type: text/markdown response header and a Vary: Accept header so any cache stores each representation separately.
Standard HTTP content negotiation differs from ‘cloaking’. Cloaking serves different content to deceive a crawler. Markdown negotiation serves the same content in a cleaner wrapper, by explicit request, which the spec has allowed for decades.
Measuring the token tax
Every HTML response carries structure an agent must strip before it can read the words. For markdown conversion, it costs computation, latency, and tokens.
| Content type | HTML tokens | Markdown tokens | Reduction |
## About Us heading | 12–15 | ~3 | ~75% |
| Cloudflare blog post | 16,180 | 3,150 | 80% |
| Read the Docs page | higher | ~1,496 | varies |
Which AI agents request markdown today?
A February 2026 review found 3 of 7 major agents sending a markdown preference: Claude Code, Cursor, and OpenCode. The others default to */*, accepting anything and converting HTML themselves.
Cloudflare confirms it already sees Claude Code and OpenCode requesting markdown across its network.
Cloudflare’s markdown conversion runs on its network, and builders on Workers AI can read the x-markdown-tokens header to size a context window or set a chunking strategy.
Caching needs care: Vary: Accept keeps the HTML and markdown versions distinct, at the cost of a lower cache hit rate per URL.
How to enable markdown negotiation on WordPress
Cloudflare, no code
If your site runs behind Cloudflare, enable Markdown for Agents in the dashboard. The network converts any HTML page to markdown on request, and adds a content-signal header declaring how the content can be used.
A plugin, one toggle
Simple Agent Kit reads the Accept: text/markdown header and returns clean markdown responses for every post and page. The plugin handles the header detection, the response type, and the discovery endpoints, with no separate markdown URLs to maintain.
Custom, for headless setups
Write a rewrite rule that inspects the Accept header and routes markdown requests to a dedicated handler, converting your stored content to markdown on the fly.
Markdown negotiation is one layer of an agent-ready stack that also includes llms.txt, content signals, and agent skills discovery.
The standards are converging, and the sites that adopt them now will serve the next wave of AI platforms without rework.
Simple Agent Kit is free. Get it on GitHub.