Markdown for Agents on the Cloudflare Free Plan

Published: Feb 14, 2026

Keywords: ai, webdev

In February 2026, Cloudflare announced Markdown for Agents↗, a feature that automatically converts HTML pages to Markdown when AI agents request it via Accept: text/markdown. It’s a great idea, but it requires a Pro plan or higher. And even if you have one, the edge conversion includes your navigation chrome, footers, and other layout elements that aren’t useful to an agent.

If you author content in Markdown already (as most static site generators do), you can serve curated Markdown directly from your origin and get better results. Here’s how I set this up for my Astro site behind Cloudflare’s free plan, including a caching trick that took some digging to figure out.

The Idea
Build: Generating Markdown
Serve: Caddy Content Negotiation
Cache: The Cloudflare Free Plan Problem
The Fix: Transform Rules
Results
Trade-offs vs. Edge Conversion
What’s Next

The Idea

AI coding tools like Claude Code and OpenCode already send Accept: text/markdown when fetching URLs. The Markdown for Agents spec↗ defines the protocol: the client sends an Accept header, and the server responds with clean Markdown plus a Content-Signal header indicating how the content may be used.

Since my blog posts and guides are authored in MDX, the source Markdown is already available at build time. Rather than converting HTML back to Markdown at the CDN edge, I can generate purpose-built Markdown files during the Astro build and serve them from the origin when an agent asks for them.

Build: Generating Markdown

During astro build, a post-build script walks the content collections (blog posts and guides), strips MDX-specific syntax like JSX components and import statements, and writes clean Markdown files to a dist-md/ directory that mirrors the URL structure of the HTML site.

Each generated file gets proper YAML frontmatter with the title, publication date, and description, followed by the content body. For index pages (the blog listing, guide listing, homepage), I maintain template files with an  marker that gets replaced with a sorted, linked listing of entries during generation.

The result is two parallel directory trees in the build output:

dist/ — the regular HTML site
dist-md/ — Markdown variants for agents

Serve: Caddy Content Negotiation

Both directories are served from the same container. Caddy handles the content negotiation with a named matcher that checks for Accept: text/markdown and verifies a Markdown file exists for the requested path:

@wantsMarkdown {
    header Accept *text/markdown*
    file {
        root /srv/site/dist-md
        try_files {path}index.md {path}/index.md
    }
}
handle @wantsMarkdown {
    root * /srv/site/dist-md
    header Content-Type "text/markdown; charset=utf-8"
    header Content-Signal "ai-input=yes, search=yes"
    header Cache-Control "public, max-age=3600, s-maxage=86400"
    header Vary Accept
    rewrite {file_match.relative}
    file_server
}

If no Markdown file exists for a given path, the file matcher fails and the request falls through to the default HTML handler. This means agents get Markdown for content pages and HTML for everything else, with no 404s or broken behavior.

You can test it with curl:

# HTML (default)
curl -sI https://mwolson.org/blog/

# Markdown
curl -sI -H 'Accept: text/markdown' https://mwolson.org/blog/

Cache: The Cloudflare Free Plan Problem

This is where it gets interesting. Caddy sends Vary: Accept on Markdown responses, which should tell any standards-compliant cache to store separate variants keyed on the Accept header. Cloudflare’s free plan doesn’t do this. It caches a single variant per URL regardless of Vary.

The practical consequence: if an HTML response gets cached first, subsequent Markdown requests return the cached HTML. If a Markdown response gets cached first, regular browsers get Markdown. Neither is acceptable.

My first workaround was setting s-maxage=0 on Markdown responses to prevent Cloudflare from caching them. This solved the correctness issue but meant every Markdown request hit the origin, defeating the point of having a CDN.

The Fix: Transform Rules

The solution uses Cloudflare’s Transform Rules↗, which are available on the free plan (up to 10 rules). Transform Rules run at phase 3 in Cloudflare’s request pipeline, well before cache lookup at phase 19.

The rule appends a query parameter to requests that include Accept: text/markdown, which gives them a different cache key from regular HTML requests:

Rule name: Markdown for Agents cache key
Filter expression: any(http.request.headers["accept"][*] contains "text/markdown")
Query rewrite (static): _fmt=md

With this rule active, a request to /blog/ with Accept: text/markdown becomes /blog/?_fmt=md internally before cache lookup. Cloudflare now stores two cache entries: one for the HTML variant and one for the Markdown variant. The origin may see the ?_fmt=md query parameter, but Caddy matches on the Accept header as before, so no origin changes are needed.

Once I confirmed this was working, I raised s-maxage on Markdown responses to match the HTML caching policy. Both variants are now edge-cached with a 1-day TTL.

Results

After deploying and purging the Cloudflare cache, both variants cache independently:

# First HTML request
content-type: text/html; charset=utf-8
cf-cache-status: MISS

# Second HTML request
content-type: text/html; charset=utf-8
cf-cache-status: HIT

# First Markdown request
content-type: text/markdown; charset=utf-8
cf-cache-status: MISS

# Second Markdown request
content-type: text/markdown; charset=utf-8
cf-cache-status: HIT

The Markdown responses include Content-Signal: ai-input=yes, search=yes as specified by the Markdown for Agents protocol, and the content is clean Markdown without navigation elements, script tags, or layout chrome.

Trade-offs vs. Edge Conversion

There are real trade-offs to the origin approach compared to Cloudflare’s edge conversion:

You have to maintain a build step that generates the Markdown files. If your content changes frequently or comes from a CMS, this adds complexity.
You need to handle content negotiation at the origin (Caddy, NGINX, or similar).
The free plan Transform Rule trick uses one of your 10 available rules.

On the other hand:

You control exactly what the Markdown output looks like. No navigation chrome, no footer links, no cookie banners converted to Markdown.
The Markdown can include proper YAML frontmatter with structured metadata that agents can parse.
It works on any Cloudflare plan, including free.
For static sites where the content is already in Markdown, the build step is straightforward.

What’s Next

I might extract the build script into a reusable Astro integration so that other Astro sites can add origin-level Markdown for Agents support without writing custom build scripts. If I do, that will be the subject of a future post.

Contents