# Serving Markdown to AI Agents with CloudFront Functions and SST
> A deep dive into Accept-header content negotiation on a fully static site – CloudFront Functions, SST edge injections, and the gotchas I hit along the way.
Published: 2026-08-12
Tags: ai, aws, sst
Canonical: https://wempe.dev/blog/serving-markdown-to-ai-agents-with-cloudfront
In [my last post on optimizing a blog for AI](/blog/optimizing-your-blog-for-ai), I claimed that every post on this site is available as plain markdown – append `.md` to any post URL, or send an `Accept: text/markdown` header to the regular URL:
```bash
curl -H "Accept: text/markdown" https://wempe.dev/blog/serving-markdown-to-ai-agents-with-cloudfront
```
I also promised the implementation is a post of its own. This is that post.
The interesting part is not generating markdown – that's easy. The interesting part is that this blog is **fully static**: prerendered HTML served from S3 behind CloudFront. There is no server that could look at an `Accept` header and decide what to send. So the content negotiation has to happen at the edge – in a CloudFront function, with a 10KB size limit, wired up through SST.
Here's the complete setup, including the gotchas that were not obvious to me going in.
## What We're Building
Two representations of every post, one canonical URL:
- `GET /blog/{slug}` → HTML (browsers)
- `GET /blog/{slug}.md` → markdown (explicit)
- `GET /blog/{slug}` with `Accept: text/markdown` → markdown (content negotiation)
The negotiated case is the one that needs infrastructure. This is the request flow:
```mermaid
sequenceDiagram
participant A as AI agent
participant F as CloudFront function
participant C as CloudFront cache / S3
A->>F: GET /blog/my-post Accept: text/markdown
F->>F: Accept prefers markdown? → rewrite URI to /blog/my-post.md
F->>C: GET /blog/my-post.md
C->>A: 200 text/markdown Link: rel="canonical" Vary: Accept
```
Why negotiate at all instead of just offering the `.md` URLs? Because agents don't know about them. An agent following a link from search results, an llms.txt file, or a user's pasted URL lands on the canonical HTML URL. Content negotiation is the mechanism HTTP already has for exactly this situation – the client states what it can process, the server picks the best representation.
## Step 1: A Markdown Twin for Every Post
The markdown itself is generated at build time by a plain Astro endpoint. `src/pages/blog/[id].md.ts` prerenders one `.md` file per post:
```ts
// src/pages/blog/[id].md.ts
import { getCollection } from 'astro:content';
import type { APIRoute, GetStaticPaths } from 'astro';
import { renderBlogPostMarkdown } from '~/lib/blog-markdown';
export const prerender = true;
export const getStaticPaths = (async () => {
const posts = await getCollection('blog');
return posts.map((post) => ({ params: { id: post.id }, props: { post } }));
}) satisfies GetStaticPaths;
export const GET: APIRoute = ({ props }) => {
const post = props.post;
const markdown = renderBlogPostMarkdown(post, canonicalUrl(post));
return new Response(markdown, {
headers: {
// Only effective in dev; in prod the prerendered file is served
// from S3 and CloudFront sets the headers (more on that below).
'Content-Type': 'text/markdown; charset=utf-8',
},
});
};
```
`renderBlogPostMarkdown` takes the raw MDX body from the content collection and turns it into standalone markdown: it strips the `import` statements, converts each MDX component into a markdown equivalent (my `> [!NOTE]
> ` becomes a GitHub-style `> [!NOTE]` blockquote, embeds become links), rewrites relative image paths to absolute URLs, and prepends the metadata an LLM would otherwise scrape from the page – title, dates, tags, and the canonical URL.
>
> The HTML page advertises its markdown twin in the ``, so the relationship is discoverable in both directions:
>
> ```html
>
> ```
>
> At this point `.md` URLs work. On to the part the post title promised.
>
> ## Why Rewrite at the Edge Instead of Redirecting?
>
> My first instinct was a redirect: agent asks for markdown, respond with `302` to the `.md` URL. It works, but a rewrite is better on every axis I care about:
>
> - **One round-trip instead of two.** Agents fetching your content pay the latency; some won't follow the redirect at all.
> - **The canonical URL stays the URL.** The agent asked for `/blog/{slug}` and gets content for `/blog/{slug}` – just in the representation it asked for. That's how content negotiation is supposed to work.
> - **No signal fragmentation.** There's only one URL being linked, cited, and indexed. The markdown response carries a `Link: <...>; rel="canonical"` header pointing back to the HTML URL, so even a crawler that fetches the `.md` file directly knows where the canonical lives.
>
> So instead of redirecting, the CloudFront function **rewrites the request URI** before CloudFront does anything else with it: `/blog/{slug}` becomes `/blog/{slug}.md`, and S3 serves the prerendered markdown file as if that's what was asked for all along.
>
> ## Why CloudFront Functions and Not Lambda@Edge?
>
> CloudFront gives you two options for running code at the edge. For this use case, [CloudFront Functions](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/cloudfront-functions.html) win clearly over Lambda@Edge:
>
> - They run in **sub-millisecond** time directly at the edge location, with no cold starts.
> - They are basically **free at blog scale** (2 million invocations per month included, $0.10 per million after).
> - They can attach to the **viewer request** phase, which runs *before* the cache lookup – this matters a lot for caching, as we'll see.
>
> The trade-off is a heavily constrained runtime: a JavaScript subset, no network or filesystem access, and a **10KB total size limit** per function. No npm packages – every line is handwritten. For parsing a header and rewriting a URI, that's fine. The 10KB limit will come back to bite us in the SST section though.
>
> ## The Viewer-Request Function
>
> The rewrite logic itself is small. If the request is a `GET`/`HEAD` for a blog post URL and the `Accept` header prefers markdown over HTML, rewrite the URI:
>
> ```js
> // infra/cloudfront/markdown-negotiation-viewer-request.js
> function negotiateMarkdownRequest(request) {
> var method = request.method;
> var slug = getBlogPostHtmlSlug(request.uri);
> if ((method === "GET" || method === "HEAD") && slug && prefersMarkdown(getHeaderValue(request.headers.accept))) {
> request.uri = "/blog/" + slug + ".md";
> }
> }
>
> // Single source of truth for "which URIs are blog posts".
> function getBlogPostHtmlSlug(uri) {
> var match = uri.match(/^\/blog\/([^\/.]+)(?:\/|\/index\.html)?$/);
> return match ? match[1] : "";
> }
> ```
>
>
> The URI regex is a contract that spans layers: my content collection schema constrains slugs to kebab-case (`/^[a-z0-9]+(?:-[a-z0-9]+)*$/`) precisely so that a slug can never contain a dot or slash that would break this matcher. The constraint is documented in both places, pointing at each other.
### Parsing the Accept Header Properly
The tempting shortcut is `accept.includes("text/markdown")`. Don't. The `Accept` header has semantics – quality values, wildcards, precedence rules – and clients use them. A few headers that a substring check gets wrong:
| `Accept` header | Substring check | Correct answer |
| --- | --- | --- |
| `text/markdown;q=0` | markdown | HTML (q=0 means "not acceptable") |
| `text/markdown;q=0.9, text/html` | markdown | HTML (html has implicit q=1) |
| `text/markdown;q=0.5, text/*;q=0.8` | markdown | HTML (exact type beats wildcard, and its q is lower) |
| `text/html;q=0.8, text/markdown` | markdown | markdown ✓ (correct, by accident) |
So `prefersMarkdown` does real [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110#name-accept) negotiation, condensed to what this decision needs:
1. **Parse** the header into media ranges with their q-values (respecting quoted parameters – a comma inside `;foo="a,b"` is not a separator).
2. **Require an explicit `text/markdown`** with q > 0. Wildcards like `*/*` or `text/*` never opt a client into markdown – a browser sending `text/html,application/xhtml+xml,...,*/*;q=0.8` must get HTML.
3. **Compare the best matching preference** for `text/markdown` vs. `text/html`, where the most specific matching range determines the q-value (exact type > type wildcard > full wildcard), per RFC 9110 precedence.
That's about 100 lines of dependency-free ES5-style JavaScript in `accept-negotiation.js`. Tedious, but it's exactly the kind of pure logic that's trivial to unit test – more on that below.
## The Viewer-Response Function
Rewriting the request is only half the job. The markdown file is served from S3, and the response needs three headers S3 won't reliably provide:
- `Content-Type: text/markdown; charset=utf-8` – so clients don't guess the encoding.
- `Link: ; rel="canonical"` – the HTTP-header equivalent of ``, which an HTML page can carry but a markdown file cannot.
- `Vary: Accept` – on **both** representations. Any cache between CloudFront and the client must know that this URL's response depends on the `Accept` header, otherwise a shared cache could serve markdown to a browser (or vice versa).
So a second CloudFront function runs on the viewer-response phase:
```js
// infra/cloudfront/markdown-negotiation-viewer-response.js
function negotiateMarkdownResponse(request, response) {
var blogPost = getBlogPostRepresentation(request);
if (!blogPost) return;
addVary(response.headers, "Accept");
if (blogPost.format === "markdown") {
response.headers["content-type"] = { value: "text/markdown; charset=utf-8" };
response.headers.link = {
value: "; rel=\"canonical\"",
};
}
}
```
### The Gotcha: Your Rewrite Is Invisible to the Response Function
Here's the part that cost me the most time. My first version of `getBlogPostRepresentation` just checked whether `request.uri` ends in `.md` – after all, the viewer-request function rewrote it, right?
Wrong. The `event.request` object a viewer-response function receives is **not guaranteed to reflect modifications made by the viewer-request function**. In my testing it showed the *original* URI – `/blog/{slug}`, no `.md` in sight – and the response function happily skipped the header fixes.
The fix: the response function **re-runs the negotiation** instead of trusting the URI. It matches both shapes – a direct `.md` request *and* an HTML-shaped URI whose `Accept` header prefers markdown – and re-derives the same decision the request function made:
```js
function getBlogPostRepresentation(request) {
if (request.method !== "GET" && request.method !== "HEAD") return null;
var markdownSlug = getBlogPostMarkdownSlug(request.uri);
if (markdownSlug) return { slug: markdownSlug, format: "markdown" };
var htmlSlug = getBlogPostHtmlSlug(request.uri);
if (!htmlSlug) return null;
return {
slug: htmlSlug,
format: prefersMarkdown(getHeaderValue(request.headers.accept)) ? "markdown" : "html",
};
}
```
This is also why the `Accept` parser and the URI matchers live in their own files: both functions need the exact same logic, because **they must reach the same decision for the same request**. If the request function rewrites but the response function doesn't recognize it, you serve markdown with broken headers.
## Wiring It Up with SST
This blog is deployed with [SST](https://sst.dev/), and its `Astro` component already creates the whole CloudFront distribution – including its **own** CloudFront function that handles routing. You can't simply attach a second function to the same event (CloudFront allows one function per event type per behavior). Instead, SST exposes an [`edge.viewerRequest.injection`](https://sst.dev/docs/component/aws/astro/#edge) option: a string of JavaScript that gets injected *into* SST's function handler.
That has two consequences. First, the injected code shares scope with SST's handler, where the `event` object lives – so my files define pure functions, and the call sites are appended in `infra/www.ts`:
```ts
// infra/www.ts
const sharedNegotiationInjection =
readCloudFrontInjection("accept-negotiation.js") + readCloudFrontInjection("blog-post-uri.js");
const viewerRequestInjection =
readCloudFrontInjection("markdown-negotiation-viewer-request.js") +
"negotiateMarkdownRequest(event.request);" +
sharedNegotiationInjection;
const viewerResponseInjection = (
readCloudFrontInjection("markdown-negotiation-viewer-response.js") +
"negotiateMarkdownResponse(event.request,event.response);" +
sharedNegotiationInjection
).replaceAll("__SITE_ORIGIN__", `https://${hostname}`);
export const astro = new sst.aws.Astro("Astro", {
path: "apps/www",
edge: {
viewerRequest: { injection: viewerRequestInjection },
viewerResponse: { injection: viewerResponseInjection },
},
// ...
});
```
Second, remember the **10KB size limit**? The injection shares it with SST's own routing code. A hundred lines of RFC-compliant header parsing with comments eats into that budget fast. So `readCloudFrontInjection` minifies each file at deploy time with esbuild – the same thing SST does with its own code:
```ts
function readCloudFrontInjection(fileName: string) {
const code = readFileSync(join($cli.paths.root, "infra", "cloudfront", fileName), "utf8");
return transformSync(code, {
minifyWhitespace: true,
minifySyntax: true,
target: "es2018",
}).code;
}
```
The source files stay readable and commented; the deployed function gets the compact version.
## What About Caching?
This is where the viewer-request phase quietly earns its keep. CloudFront functions on viewer request run **before the cache lookup** – so the rewritten URI *is* the cache key.
Think about what that means: a negotiated request for `/blog/{slug}` with `Accept: text/markdown` and a direct request for `/blog/{slug}.md` hit **the same cache entry**. There is no need to add the `Accept` header to the cache key (which would fragment the cache across every distinct `Accept` string browsers send), and no risk of cache poisoning where a markdown response gets cached under the HTML URL. The two representations are two cache entries, keyed by two URIs, and the function deterministically routes every request to the right one.
You can watch it happen:
```bash
$ curl -sI https://wempe.dev/blog/optimizing-your-blog-for-ai.md | grep -iE "etag|x-cache"
etag: "918b11804ae27d8917a5d384c1eefa0f"
x-cache: Miss from cloudfront
$ curl -sI -H "Accept: text/markdown" https://wempe.dev/blog/optimizing-your-blog-for-ai | grep -iE "etag|x-cache"
etag: "918b11804ae27d8917a5d384c1eefa0f"
x-cache: Hit from cloudfront
```
Same ETag, and the negotiated request is a cache **hit** off the entry the direct `.md` request just created.
## Testing CloudFront Functions Without Deploying
Edge functions have a miserable feedback loop if your test strategy is "deploy and curl". But since the negotiation logic is pure functions in plain files, it can be tested with `node:test` directly – no CloudFront, no mocks of AWS, no build step.
The test file evaluates the **actual source files** – the same bytes that get minified and injected at deploy time – and reconstructs the same glue `www.ts` appends:
```js
// infra/cloudfront/markdown-negotiation.test.mjs
const read = (name) => readFileSync(join(import.meta.dirname, name), "utf8");
const shared = read("accept-negotiation.js") + read("blog-post-uri.js");
const negotiateMarkdownRequest = new Function(
`${read("markdown-negotiation-viewer-request.js") + shared}; return negotiateMarkdownRequest;`,
)();
```
Most of the suite is a table of `Accept` headers and expected outcomes – including the adversarial ones from the table above, quoted-parameter commas, invalid q-values, and the real-world headers browsers and AI agents actually send. When I got the RFC 9110 precedence rules wrong on the first attempt (wildcard q-values must *not* override an exact match), a table row caught it, not a production curl.
> [!WARNING]
> One thing unit tests can't catch: the viewer-response gotcha above. That the response function sees the original URI is CloudFront runtime behavior – no local test would have surfaced it. For the integration-level behavior, a post-deploy `curl` check against the live distribution is still part of my routine.
## Conclusion
Content negotiation on a static site sounds like a contradiction – negotiation needs a server, static sites don't have one. CloudFront Functions resolve it neatly: a URI rewrite before the cache lookup turns "negotiate per request" into "pick one of two prerendered files", at sub-millisecond latency and effectively zero cost.
The pieces that made it robust rather than just working:
- **Rewrite, don't redirect** – one round-trip, one canonical URL.
- **Parse `Accept` for real** – q-values, wildcards, and RFC 9110 precedence; a substring check serves markdown to clients that said `q=0`.
- **Don't trust the rewritten URI in viewer-response** – re-run the negotiation from shared, single-source-of-truth helpers.
- **`Vary: Accept` + `Link: rel="canonical"`** – keep downstream caches correct and search signals consolidated.
- **Test the source files directly** – the edge runtime is constrained, but the logic is just functions.
Whether markdown representations actually increase AI citations remains unproven – I graded that claim honestly [in the previous post](/blog/optimizing-your-blog-for-ai). But the mechanical argument stands: agents that fetch your content get a representation that's cheaper and more faithful to ingest, from the same URL everyone else links to. And now you know exactly what that costs to build: two small edge functions, one Astro endpoint, and a healthy respect for the `Accept` header.
## TLDR
- Every post is prerendered twice: HTML page + `.md` twin from an Astro endpoint (`prerender = true`), both static on S3.
- A CloudFront **viewer-request function** rewrites `/blog/{slug}` to `/blog/{slug}.md` when the `Accept` header genuinely prefers markdown – full q-value/wildcard/precedence parsing, not a substring check.
- Rewriting before the cache lookup makes the rewritten URI the cache key: negotiated and direct `.md` requests share one cache entry, no `Accept` in the cache key needed.
- A **viewer-response function** sets `Content-Type`, a canonical `Link` header, and `Vary: Accept` – and must re-run the negotiation, because viewer-request URI rewrites are not visible in the viewer-response event.
- SST wires it up via `edge.*.injection` into its own router function; the shared 10KB CloudFront function limit is managed by minifying the injected sources with esbuild at deploy time.
- The negotiation logic is pure, dependency-free functions – unit-tested with `node:test` against the exact source files that get deployed.
## Sources
- [CloudFront Functions overview](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/cloudfront-functions.html) – submillisecond startup, scaling, use cases
- [Choosing between CloudFront Functions and Lambda@Edge](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/edge-functions-choosing.html) – the full comparison table: duration, scale, 10KB size limit, runtime restrictions
- [CloudFront pricing](https://aws.amazon.com/cloudfront/pricing/) – 2 million function invocations per month free, $0.10 per million after
- [RFC 9110: HTTP Semantics](https://www.rfc-editor.org/rfc/rfc9110) – proactive content negotiation (§12.1), quality values (§12.4.2), `Accept` precedence (§12.5.1), `Vary` (§12.5.5)
- [SST `Astro` component – `edge` option](https://sst.dev/docs/component/aws/astro/#edge) – `viewerRequest.injection` / `viewerResponse.injection` and how the code is injected into SST's own handler