AWS Builder Center

Builder Center

Content Negotiation at the Edge with Amazon CloudFront and AWS WAF

Content Negotiation at the Edge with Amazon CloudFront and AWS WAF

Your CDN can do more than cache - it can detect what each client needs and serve optimized content formats automatically, with zero origin changes and zero risk.

Eitav Arditti

The web serves more than browsers now

A growing share of web traffic doesn't come from browsers. AI agents crawl your pages to build knowledge. API consumers pull structured data for integrations. Accessibility tools parse your content for screen readers. CLI utilities fetch documentation for developers working in terminals. Yet every one of these clients receives the same response: full HTML pages with navigation bars, JavaScript bundles, cookie consent banners, and tracking pixels.
This post explores edge-native content negotiation: your CDN classifies each request, checks whether an optimized format already exists, and routes to it - all in under a millisecond, with automatic failover to original content if anything is missing. You keep a single source of truth (your HTML). The edge handles the rest. No origin changes. No new endpoints. No risk.
The pattern is generic - any client type, any output format - but we make it concrete with a working sample: Amazon CloudFront and AWS WAF detect AI bots that send  Accept: text/markdown , serve pre-generated markdown when available, and trigger background conversion when not.

Why not existing approaches?

HTTP content negotiation via  Accept  headers has existed since the earliest days of the protocol. In practice, almost nobody implements it at the CDN layer. Servers return HTML regardless of what the client asked for.
Static workarounds like  llms.txt  have emerged as a way to point AI agents at machine-readable content. But these files are manual, go stale quickly, and don't scale across thousands of pages. They put the burden on the client to discover a map rather than receiving what it needs directly.
What's been missing is a way to negotiate formats at the edge - where you already have request classification, global state, and programmable routing. The building blocks exist today.

Edge-native content negotiation

Three capabilities, available now in Amazon CloudFront and AWS WAF, make this pattern possible:
  1. Classification: AWS WAF as a request classifier
AWS WAF does more than block threats. Its managed rule groups - particularly  AWS WAF Bot Control  - classify incoming requests and apply labels. An AI bot request gets the label  awswaf:managed:aws:bot-control:bot:category:ai . A verified search engine crawler gets a different label. You can write rules that match these labels and inject custom headers into the request before it reaches your origin.
This turns AWS WAF into a request classifier. A single rule can say: "If this request is from an AI bot and its  Accept  header includes  text/markdown , inject a header telling downstream logic that format conversion is needed." No application code changes. No User-Agent string parsing. The classification happens at the edge, using a managed ruleset that AWS maintains and updates as new bots appear.
  1. Sub-millisecond state: CloudFront KeyValueStore
Amazon CloudFront KeyValueStore (KVS) is a globally distributed key-value store accessible from CloudFront Functions. A function can look up a key - "does a markdown version of  /docs/architecture.html  exist?" - and get a response in under 1 millisecond. This lookup happens before the cache lookup, so routing decisions can be made on every request without adding latency.
KVS holds lightweight markers, not the content itself. A marker says: "A markdown version exists at this path, it's valid until this timestamp." The actual content lives in your origin (Amazon Simple Storage Service (Amazon S3), Application Load Balancer, or any HTTP server).
  1. Dynamic routing: CloudFront Functions
CloudFront Functions run JavaScript at every CloudFront edge location. They execute on every request with sub-millisecond overhead. A viewer-request function can read the classification headers injected by AWS WAF, look up KVS to check whether an optimized format exists, and route the request accordingly - including creating a dynamic origin group with failover for safe fallback.

The flow

Put these together and every request follows a clear path:
  1. Classify - AWS WAF labels the request and injects headers
  1. Check - CloudFront Function reads the header, queries KVS for a format marker
  1. Route - If a marker exists and is valid, rewrite the request to the formatted content path. If not, serve the original content.
  1. Fail over - If the formatted content doesn't exist at the expected path, a dynamic origin group fails over to original content automatically
Sub-millisecond. No origin changes.
A key mitigation built into the design: every failure mode falls back to original content. If the optimized format hasn't been generated yet, the client gets HTML - exactly what it would have received without this system. If KVS is unavailable, original content. If a marker has expired, original
content-negotiation-flow

Request classification, KVS lookup, and dynamic routing happen at the edge. The origin group failover path ensures clients always receive content, even when the optimized format hasn't been generated yet.

The sample: AI bots and markdown

To make this pattern concrete, we built a  sample application  that serves markdown to AI bots. It's deployed with a single  npx cdk deploy  command and demonstrates the full lifecycle: detection, routing, generation, and cleanup.

Detection with content negotiation

The sample uses AWS WAF Bot Control (COMMON inspection level, Version 5.0) to identify AI bots. But identification alone isn't enough - not every AI bot wants markdown. The WAF rule uses an AND statement that requires both an AI bot category label and an  Accept  header containing  text/markdown :
// WAF AND rule: both conditions must match
andStatement: {
  statements: [
    {
      labelMatchStatement: {
        scope: 'LABEL',
        key: 'awswaf:managed:aws:bot-control:bot:category:ai',
      },
    },
    {
      byteMatchStatement: {
        searchString: 'text/markdown',
        fieldToMatch: { singleHeader: { name: 'accept' } },
        positionalConstraint: 'CONTAINS',
        textTransformations: [{ priority: 0, type: 'LOWERCASE' }],
      },
    },
  ],
},
This is proper content negotiation. A bot that sends  Accept: text/html  gets HTML. A bot that sends  Accept: text/markdown, text/html  gets markdown when available, HTML otherwise. The WAF rule injects a  format-convert-needed: markdown  header only when both conditions match.

Routing and failover

The CloudFront Function reads this header and checks KVS. Three outcomes are possible:
  • Marker found, valid TTL: The function creates a dynamic origin group and rewrites the request URI to the formatted content path (e.g.,  /formats/markdown/docs/architecture.html ). The response includes YAML frontmatter with title, description, and token count, plus  x-amz-meta-*  headers for programmatic access.
  • Marker not found: The function serves original content and emits a  format_needed  signal via  console.log() .
  • Marker expired: The function serves original content and signals re-generation.
In every case, the client receives a valid response.

Asynchronous generation

The generation pipeline turns a "format needed" signal into a stored, routable format - without blocking any request.
Async-convertion-architecture

The generation pipeline converts CFF log signals into stored formats. SQS FIFO deduplication means 100 simultaneous requests for the same page trigger only one generation.
The CloudFront Function's  console.log()  output flows to Amazon CloudWatch Logs. A subscription filter routes matching events to a Parser AWS Lambda function, which sends messages to an Amazon Simple Queue Service (Amazon SQS) FIFO queue. The FIFO queue deduplicates by URI - if 100 bots request the same page within a 5-minute window, only one generation job runs.
The Generator Lambda function fetches the original HTML through CloudFront (using an internal header to bypass the conversion logic), converts it to GitHub Flavored Markdown using  turndown , extracts metadata from  <head>  elements, adds YAML frontmatter, and writes the result to Amazon S3 with  x-amz-meta-*  headers. It also writes a marker to Amazon DynamoDB with a 24-hour TTL.
An Amazon DynamoDB Streams trigger fires the KVS Sync Lambda function, which writes the marker to CloudFront KVS and invalidates any stale cached content at the format path. On TTL expiry, the same function removes the KVS key and the S3 format file. The next bot request triggers regeneration - keeping content fresh with zero manual maintenance.

The pattern is extensible

The sample uses AI bots and markdown, but the pattern itself is format-agnostic and signal-agnostic.
New formats. Adding a format means implementing a single interface:
export interface Converter {
  contentType: string;
  convert(html: string): ConvertResult;
}
Plaintext for CLI tools. JSON-LD for structured data consumers. Simplified HTML for accessibility readers. Each converter plugs into the same pipeline. The edge routing - classification, KVS lookup, dynamic origin group - works unchanged.
New signals. The classification layer is a WAF rule. Want to serve optimized images to mobile clients? Add a rule that matches device-type headers. Regional content preferences? Match geographic labels. API consumers that send  Accept: application/json ? Same pattern. The WAF rule is the only thing you change - the rest of the system adapts.
Any origin. The sample uses Amazon S3 as its origin, but the pattern works with any HTTP server behind CloudFront: Application Load Balancer, Amazon API Gateway, an on-premises server, or even another CDN. You can place your existing content delivery setup behind CloudFront as a custom origin and add format negotiation as a standalone layer - without migrating anything.
Format-aware caching. The WAF-injected header is included in the CloudFront cache policy's cache key. This means the same URL maintains separate cache entries for different client types - a browser and an AI bot hitting the same URL get independent cached responses. A  Vary: Accept  response header ensures downstream proxies and intermediaries respect the content negotiation.

Try it

Deploy the sample in under 5 minutes:
$ git clone https://github.com/aws-samples/sample-automated-markdowns-for-agents-using-cloudfront-and-waf.git
$ cd sample-automated-markdowns-for-agents-using-cloudfront-and-waf
$ npm install
$ npx cdk deploy
Then watch the negotiation in action with three requests:
# 1. Browser request - returns HTML
$ curl -s https://<DISTRIBUTION_DOMAIN>/blog/getting-started.html | head -5

# 2. AI bot request - first time returns HTML, triggers async generation
$ curl -H "User-Agent: GPTBot/1.0" -H "Accept: text/markdown, text/html" \
    https://<DISTRIBUTION_DOMAIN>/blog/getting-started.html

# 3. Wait ~30 seconds, repeat - returns markdown with YAML frontmatter
$ curl -D - -H "User-Agent: GPTBot/1.0" -H "Accept: text/markdown, text/html" \
    https://<DISTRIBUTION_DOMAIN>/blog/getting-started.html
The third request returns structured markdown:
Content-Type: text/markdown; charset=utf-8
x-amz-meta-title: Getting Started - CloudFront AI Format Converter
x-amz-meta-description: Deploy and test the automated format conversion solution...
x-amz-meta-tokens: 1915
Vary: Accept

---
title: Getting Started - CloudFront AI Format Converter
description: Deploy and test the automated format conversion solution for AI agents in under 15 minutes.
tokens: 1234
---

# Getting Started
...
Resources:

Cleanup

The sample creates billable AWS resources. To tear down the stack:
$ npx cdk destroy
This removes all resources created by the stack, including the CloudFront distributions, S3 buckets, Lambda functions, DynamoDB table, and WAF web ACLs.

Conclusion

We can't predict what the next client type will be, or what content format it will need. But we know that adaptation has to be straightforward - something you extend at the edge, not something that forces you to rewrite your origin.
With this pattern, you keep a single source of truth: your original content. You focus on what you do best - creating and maintaining that content. When new actors arrive - AI agents today, something else tomorrow - you add a WAF rule, implement a converter, and the edge handles the rest. No origin changes.
 Deploy the sample , extend it with your own formats and signals, and be ready for whatever comes next.
# amazon-cloudfront# aws-waf# waf# networking-content-delivery# markdown
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article