Why Nodex serves Markdown to AI agents

A growing share of the traffic to the recruitment sites we host is an AI agent, not a person. Each one downloads a full HTML page to read a few hundred words of text. This is why we now serve markdown over the same URL using content negotiation, instead of the llms.txt file the industry tried first, and the two problems we ran into on the way: a caching mistake and a denial-of-service in the Accept header.

Look at your access logs and you'll see a pattern that wasn't there two years ago. A large share of the requests to a recruitment site are no longer a candidate browsing jobs or a client reading an about page. They are software: a coding assistant pulling a page for context, a research tool summarising a guide, a model answering a question about an agency and its roles.

Each of those requests does the same wasteful thing. It downloads the whole HTML document, navigation and tracking scripts and cookie banner and all, then discards nearly everything to get at the few hundred words it wanted. That text was usually written in markdown in the first place, before anything rendered it to HTML for browsers. The agent would rather have the markdown back.

There are two ways to give it back. One is a new convention built for the job. The other is a mechanism that has been in HTTP since the late nineties. We used the new one first, like most people did, then replaced it with the old one.

The web already solved this in 1999

The web has always been able to serve more than one representation of the same resource, and the Accept header is how a client asks for one. The client says which formats it can handle, the server returns the best match, and the URL doesn't change. This is content negotiation, and it has been in the spec since HTTP/1.1.

An agent that wants markdown asks for it; a browser asks for what it always asks for:

GET /some-page HTTP/1.1
Accept: text/markdown
GET /some-page HTTP/1.1
Accept: text/html

One URL, two representations, and the client chooses. There's no separate index to discover and no per-tool convention to support, just the header the client already sends. And text/markdown is a registered media type (RFC 7763), so this is the mechanism working as intended rather than a workaround.

The markdown still has to exist. In our case it's generated from the same source as the page rather than maintained by hand alongside it, so it can't fall out of sync with what a visitor sees. What negotiation removes isn't the second file, it's the second address: nobody has to discover, guess or agree a convention for where that file lives.

Agents are already doing this. When Checkly tested seven coding agents in early 2026, several of the most widely used, including Claude Code and Cursor, sent Accept: text/markdown when fetching a page. Others still asked for HTML or sent a generic */*. It isn't universal yet, but the trend runs one way, and supporting it costs nothing.

This isn't really an AI feature at all

Serving text over the same URL as HTML isn't really an agent feature. It's closer to how the web should have served content all along, and agents are just the first group of clients large enough to make it worth doing.

The same representation helps anything that wants content rather than layout: reader modes, RSS and newsletter extractors, archival tools, anything on a slow or metered connection. But the reason it matters commercially right now is token cost.

Bandwidth isn't the number to look at. Gzip compresses HTML well because HTML is repetitive, the same tags and class names and navigation on every page, so on the wire the gap between HTML and markdown looks small. But a model never sees the compressed bytes. It reads the uncompressed document and pays per token, and compression does nothing there.

This article is a good example. Fetched as HTML it comes to 12,581 tokens; requested with Accept: text/markdown the same page is 3,802, roughly 70% fewer. And that's a lean, text-heavy article to begin with. The ratio grows with how much markup, script and layout a page carries relative to its words, so a homepage or a job listing built from components saves proportionally more. Stripping scripts and styles first, as a competent agent will, narrows the gap but never closes it, because what's left is the markup itself woven through the text you wanted.

So on many pages, most of what an agent pays to fetch as HTML isn't the answer it came for. Serving markdown removes that overhead.

We tried llms.txt. The data says almost nobody reads them.

So why not just publish the markdown as a separate file and point machines at it? That's what llms.txt proposes, and it was the first answer most people tried, us included. You publish a separate machine-readable file, either a single /llms.txt or a parallel tree of .md files, holding a clean copy of your content. We built the tooling, generated the files and published them. It worked.

The traffic never arrived. When Ahrefs studied 137,210 domains in mid-2026, they found that among sites publishing an llms.txt, the large majority of those files were almost never fetched, and most of the few requests that did come in were from crawlers indexing the file rather than agents reading it. Google's John Mueller had already compared the convention to the old keywords meta tag: a signal a site emits about itself, the kind search engines learned to ignore because it's easy to game. No major AI provider has committed to reading the file, and the crawlers it was meant to feed are still pulling HTML.

Two problems explain why. The first is that it rots. A file no human ever reads is a file no human ever corrects, so it drifts from the real page, and six months on an agent is reading about a service you no longer offer. The second is that nobody can find it. An agent that lands on /some-page has no reliable way to know a markdown twin exists somewhere else. Does it guess a .md suffix? Fetch /llms.txt and parse it for a map? Every tool invents its own answer, and the page author is expected to support all of them.

'But Accept doesn't solve discovery'

The strongest objection is that the two approaches solve different problems and belong together: content negotiation as the transport layer, llms.txt as the discovery layer. Accept tells the server what format to send; it says nothing about which pages exist.

That's true, but it isn't a case for llms.txt. Discovery is already solved, and has been just as long: sitemap.xml. Every page you want a machine to find, listed in a file every crawler already fetches, generated by your build, describing pages instead of duplicating them. So llms.txt doesn't reinvent one existing standard, it reinvents two. Sitemap for discovery, Accept for representation, both already working in every client and cache on the web.

Why this isn't cloaking

If you've followed this debate you'll have seen the pushback. Google's John Mueller has been dismissive of serving markdown to bots, and Microsoft's Fabrice Canel warned that dedicated bot-only pages amount to cloaking, with search engines liable to crawl both versions and compare them.

What they were objecting to is worth pinning down. The setup that prompted the warning was middleware that detects specific bot user-agents, GPTBot, ClaudeBot and the like, and serves those requests different content. That is cloaking: the server decides what you get from who it thinks you are, and no ordinary user can get the bot version.

Accept-based negotiation works the other way round. The client states a preference, not an identity. Any client can ask for either representation. Someone running curl -H 'Accept: text/markdown' gets exactly what an agent gets. Nothing is hidden and nothing is keyed to who you are, only to what you asked for. So the markdown page isn't one no user sees; it's the same page, and anyone can request it.

The one line you can't forget: Vary

Content negotiation has one sharp edge, and it's why some teams reach for a separate URL instead. As soon as a response depends on a request header, every cache in front of you (the CDN, the browser cache, any intermediary) needs to know, or it will serve the wrong representation to the wrong client.

For example: an agent requests a page with Accept: text/markdown, the cache stores the markdown against that URL, and the next person to open it in a browser gets raw markdown. Or the reverse, and the agent gets HTML.

One header fixes it:

Vary: Accept

Vary: Accept makes the Accept header part of the cache key, so the two representations are stored and served separately. It's one line, it isn't optional, and leaving it out is the most common way this breaks in production. Whenever you negotiate on Accept, set Vary: Accept on every negotiated response, the HTML one as well as the markdown one, and check your caches are honouring it.

Your Accept parser is untrusted input

To negotiate on Accept you have to parse Accept, and Accept is input the client controls. A real header from Chrome is four short entries parsed in microseconds, and nothing about it looks risky. But HTTP doesn't cap how large the header can be, and the default ceiling in most servers is around a megabyte. A parser that splits on every comma and sub-parses each entry is fine for four entries and dangerous when handed a megabyte of commas: thousands of allocations and a chunk of CPU per request before anything is served.

Now combine that with the header we just made mandatory. Vary: Accept puts a client-controlled value into the cache key, so adding junk quality values (;q=0.900001, ;q=0.900002) forces a cache miss on every request and sends the whole flood to origin. The header that makes negotiation correct is also the header that makes the attack reliable: one crafted request can bypass the cache and hit origin directly.

The fix is unglamorous. Cap the header length well below the server default, cap the number of entries you parse, and bound the split before it allocates rather than after. Treat an oversized header as no preference, which by the spec means anything is acceptable, so the client still gets a valid page. Content negotiation isn't the danger here; the header you negotiate on is untrusted input on a hot path, and it deserves the same limits as any other request field.

How it works for websites built by Nodex

We turned this on where it earns its place rather than everywhere at once. Where it's enabled:

• Pages are authored and rendered as before, so browsers get the same HTML they always got.
• The Accept header decides the response. Ask for text/markdown and you get markdown; ask for text/html and you get the normal page.
• The markdown is generated from the same source as the page, so it can't drift from what a visitor sees. That was the flaw that sank our llms.txt work.
• Vary: Accept is set on both responses and our caches respect it, so the two representations never get crossed.
• A request that names both formats without a preference falls back to HTML. That's a browser, not an agent.
• Parsing of Accept is bounded up front, so an oversized or malformed header is treated as no preference and returns HTML.

The llms.txt file can stay for anything that still looks for it, but we no longer rely on it.

None of this is new. It's the web working the way it was specified to; we just finally had a reason to use it.