Markdown for AI agents: serving readable versions of your pages

When an agent opens a page, most of what arrives is packaging: navigation, footers, scripts, tracking tags, cookie banners, layout markup. The agent has to peel all of it off to reach the article or service description it came for. It usually gets there, with a few casualties along the way: a table flattened, a heading missed, a menu mistaken for content.

Markdown takes the guesswork out. It keeps everything that carries meaning (headings, paragraphs, lists, links, tables) and drops the rest, in a compact format language models handle well. text/markdown is even a registered media type (RFC 7763), so servers and clients can name it precisely.

A word on the stakes: agents can read good HTML, and Markdown versions won't move your rankings. They're a small, inexpensive way to help machines read you more accurately, and we like small, inexpensive wins.

1. A parallel .md URL

Every page gets a twin at the same address with .md appended: /en/agency becomes /en/agency.md. The llms.txt proposal suggests exactly this convention. It's easy to test in a browser, easy to link from an llms.txt file, and safe to cache, since it's a distinct URL.

Then make the twin easy to find. Add a link in the HTML head and, if you can, an HTTP Link header on the HTML response:

<link rel="alternate" type="text/markdown" href="https://example.com/en/agency.md">

2. Content negotiation with Accept: text/markdown

HTTP has always let a client say which formats it prefers through the Accept header (RFC 9110). An agent that wants Markdown can ask for it on the page's normal URL:

curl -H "Accept: text/markdown" https://locomotive.ca/en

The server answers with Content-Type: text/markdown and the Markdown body. Browsers never notice: they ask for text/html first, so they keep receiving the page as designed. A sensible rule is to serve Markdown only when the client ranks it strictly above HTML.

Every response from a negotiated URL, HTML or Markdown, should include Vary: Accept. That header tells caches the response depends on what the client asked for.

Content negotiation has one classic trap: caching. Most CDNs and reverse proxies cache by URL. If an agent requests /en/agency as Markdown and the cache keeps that response, the next visitor may open their browser to a wall of raw Markdown where your page should be. It works the other way around too.

Vary: Accept is the standard answer, but CDNs each have their own take on it: some honour it, some ignore it, some refuse to cache any response that varies. Don't lean on it alone. Safer options:

• Bypass the cache for Markdown requests. Add a CDN rule that skips the cache when the Accept header contains text/markdown.
• Mark negotiated Markdown as private. A Cache-Control: private header tells shared caches not to store it at all.
• Cache the .md URLs normally. They're distinct addresses, so they can't collide with the HTML.

Test after every change to your CDN configuration: request a page as Markdown, load it in a browser, and confirm you get HTML.

Convert the main content only

Generate the Markdown from the page's main content area and leave the template behind, so menus and footers don't drown the substance. Clean, semantic HTML makes this almost automatic. If your CMS stores structured content, you can also render Markdown straight from it, one more reason the choice of CMS matters.

Keep it honest

The Markdown must say what the page says. Machine-readable versions that drift from what visitors see erode trust and can be treated as cloaking.

Point back to the original

• Add a small front matter block with the title, description, canonical URL and language.
• Send a Link: <page-url>; rel="canonical" header so the HTML page stays the reference.
• Send X-Robots-Tag: noindex on Markdown responses so search engines don't index duplicates.
• Keep links absolute, so they still work when the text is read out of context.

Don't forget errors

A missing page should answer in Markdown too, with the right status code and a few links to help an agent find its way: home, sitemap, llms.txt. Same for redirects: a .md request that redirects should land on a .md target.

How locomotive.ca does it

On our own site, every page has a .md version and answers Accept: text/markdown on its normal URL, for example locomotive.ca/en.md. HTML pages announce their Markdown version with a <link> tag and a Link header. Responses carry Vary: Accept; negotiated Markdown is marked private, while .md URLs are publicly cacheable. Markdown responses include front matter, a canonical link and noindex, and 404s come back as Markdown with recovery links. Each page ends with a short section on working with us, so an agent always has the context it needs. The whole story is in how we made locomotive.ca ready for AI assistants.

Start with the .md URLs: they're the simplest to build, test and cache. Add content negotiation once your caching rules are in place, then list the Markdown versions in your llms.txt file. Pair it all with structured data, and machines get both the readable text and the explicit facts.