
Serve AI crawlers a decoy maze with AWS WAF, CloudFront Functions, and Amazon Bedrock
Detect unwanted AI crawlers with AWS WAF Bot Control and swap in Bedrock-generated decoy pages at the edge — same URL, no redirect, bounded cost. A deployable sample exploring a web that renders differently per visitor.
The fourth negotiation
This is the fourth post in a series about what your CDN can negotiate on your behalf.
In Content Negotiation at the Edge with Amazon CloudFront and AWS WAF , the edge negotiated format: AWS WAF classified the client, and CloudFront served AI agents markdown instead of HTML — no origin changes.
In Monetize Any HTTP Application with x402 and CloudFront + Lambda@Edge , the edge negotiated payment: HTTP 402 and the x402 protocol turned any endpoint into a paid one.
In Per-Token Monetization of AI Traffic with AWS WAF and Amazon CloudFront , the edge negotiated value: price in proportion to what the client actually consumes.
This post takes the series to its edge case: what can the CDN negotiate with a client that refuses to negotiate at all — the crawler that ignores
robots.txt , skips the payment wall, and takes what it wants? For that visitor, the edge negotiates reality: which version of the site exists for it.Where the idea came from
Amazon MadPot is one of my favorite pieces of tech: a global fleet of honeypots — decoy services that look real to whoever probes them — that turns attacker traffic into defense intelligence at internet scale. A lot of my day-to-day work is envisioning what that idea makes possible for leading internet-scale cyber defense.
Recently I watched Adobe's approach to dynamic UI — interfaces that assemble themselves around each human, so every user consumes the same product differently.
And I thought: what if we combine the two? If we accept that two people should see two different renderings of the same page, why not extend the idea to agents? And — more interesting still — why not extend it to the agents you never invited? The same URL could serve a human the real product, a verified good bot the same thing, and an unknown or malicious agent something else entirely: a decoy rendered just for it.
This post is an exploration of that thought — an attempt to envision where the internet might go once every visitor, human or agent, gets its own rendering of reality.
Note: This sample is not intended for production use. It exists to explore an idea, not to ship one. Serving fabricated content to selected clients has legal, contractual, and security implications that depend entirely on your context — run your own legal and security reviews before taking any part of this beyond a sandbox.
A web that renders per visitor
The web was built on an assumption so old we stopped seeing it: one URL means one document, the same bytes for everyone. Fluid UIs quietly retired that assumption for humans — layouts, copy, and entire flows now assemble themselves around who is looking. Meanwhile the audience itself changed: a growing share of your traffic was never human at all. Agents read your site on someone's behalf, models train on it, and answers built from your content surface far from your domain, without a request ever coming back.
Once you put those two facts together, a different web comes into focus. Your site stops being a single artifact and becomes a policy: humans get the product, verified agents get the version you choose to give them — and the uninvited get whatever reality you decide to render for them.
That last audience is where it gets interesting, because today's toolbox for uninvited agents is binary: serve or block. Serving gives away the content and eats origin capacity for nothing in return. Blocking with a
403 is weak deterrence — it tells the crawler it was detected, so it adapts and returns. Some crawlers ignore robots.txt and every other opt-out signal, so the polite instruments don't reach them at all. Either way, the visitor decides what your infrastructure does, and you pay for it.A per-visitor web inverts that. The uninvited agent gets a third response — a mirror maze: plausible, interlinked, model-generated fiction that reflects the real site's own structure and style back at the crawler, served at the exact URL it asked for, indistinguishable from the real thing on its side of the wire. Every wall shows the crawler a convincing reflection of the site it came for, and nothing looks like a wall.

Now the economics run in your favor: the crawler spends its crawl budget on content that costs you a bounded, capped amount to produce. And because every page of that fiction is unique to you, it doubles as a tracer — when its invented phrases and URLs later surface in a model's answers, you have evidence of exactly where your content went, the visibility that today ends at your access log.
I built a deployable sample to make that future concrete: an AI maze. AWS WAF Bot Control detects the unwanted crawler, and a CloudFront Function substitutes a plausible, model-generated decoy page in place — with a
200 , no redirect, and no maze URL anywhere. Amazon Bedrock generates the decoy corpus asynchronously, never on the request path. Your origin, your visitors, and verified good bots are untouched. The idea is inspired by Cloudflare's equivalent decoy-maze feature for unwanted AI crawlers.The full sample is a single-region AWS CDK app, deployment-verified end to end. You can find it at github.com/arditti/sample-cloudfront-waf-ai-agents-maze .
The rest of this post is the concrete probe of that vision: the demo that convinced me the approach works, how the pipeline fits together, what you can observe after a test run, and how to deploy it yourself.
One URL, two realities
Both screenshots below are the same URL (
/ ) on the same deployed distribution, captured minutes apart. The only difference is who asked.A regular visitor gets the real origin content — in the sample, a seeded tea-shop page:

A flagged AI crawler gets the mirror maze: model-generated fiction with the same structure and stylesheet — a different shop, different blends, different product cards — served
200 in place, with interlinked sibling pages behind it:
There is no redirect, no maze URL, and no marker in the markup. From the crawler's side, nothing distinguishes the decoy from the real page being served. That last part is deliberate: any in-artifact identifier — a meta tag, an HTML comment, an extra JSON key — is one grep away from telling a crawler it has been mazed. Identification lives in the logs instead, which the crawler can't see.
It is not just web pages: the same trick on a JSON API
A maze made only of HTML would be a narrow trick, because a modern site is not only pages. The decoy's shape is chosen from the response content-type, not the URL, and the sample implements two archetypes:
Archetype | Chosen when the real response is | How the decoy is built |
|---|---|---|
html | text/* , application/xhtml+xml , application/xml | The real page is fetched and executed in a headless renderer, so the decoy mirrors the hydrated page rather than an empty shell |
json | application/json and the +json family ( application/ld+json , application/vnd.api+json , …) | The real payload is parsed, its schema sketched, and the fiction generated into that same schema |
Anything else is refused rather than guessed at — an image or a font gets no decoy, and the request passes through.
The sample ships a real non-S3 JSON origin to prove it: a Lambda Function URL behind the
/api/* behavior, serving a product catalogue. A visitor asking for /api/products gets the real one:{ "currency": "USD", "updated": "2026-08-04", "categories": [{ "slug": "black", "name": "Black" }], "products": [ { "sku": "HRB-FOG-100", "image": "/img/harbor-fog.svg", "batch": 4417, "name": "Harbor Fog", "category": "black", "price": 18.5, "grams": 100, "garden": "Dikom, Assam", "elevation": 980, "flush": "Second", "restedDays": 14, "mixedBy": "R.A.", "inStock": true, "notes": "Second-flush Assam cut with dried sea buckthorn from the dunes north of the harbour." } ]}A flagged crawler asking for the same URL gets a different catalogue in the same schema: different SKUs, blends, gardens and prices, with every key exactly where the real payload had it. Same rules as the HTML side —
200 , no redirect, no marker.Two constraints make the JSON case harder than HTML, and both are asserted by the test suite:
- No extra keys, ever. Adding an
x_mazemarker would make the decoy detectable with onegrepand break schema isomorphism (the property that the fake and the real payload have identical shape). The suite fails the build if such a key appears.
- The beacon has to ride inside the existing schema. HTML decoys can beacon through their own stylesheet; JSON has no renderer to exploit. So the generator repoints an existing URL-valued field — the
imagevalue above — rather than adding one. Whether the schema permits that at all depends on the payload, so the snapshot records what was done and the suite asserts against that record instead of assuming coverage.
Running
./scripts/e2e-test.sh checks this end to end: archetype detected as json from the content-type, the decoy parses as valid JSON, crawler and visitor get different catalogues, and the decoy's key set matches the real one exactly.The generalisation is the point. Any origin whose responses carry a content-type the archetypes cover — a REST API, a JSON-LD block, a server-rendered page, a hydrated single-page app — can be mazed by the same edge path, with no origin changes. What limits it is the response's media type, not what the endpoint is for.
How the pipeline works
The design has one rule I held onto throughout: never call a model on the request path. A viewer request only executes WAF classification, a CloudFront Function, a key-value lookup, and S3 delivery. Everything expensive happens asynchronously.
1. AWS WAF Bot Control classifies the request
The web ACL runs an ordered rule set: owner allowlists first, then Bot Control, then an explicit allow for verified bots, then custom user-agent, URI, honeypot, and rate classifiers. Humans and verified good bots are protected ahead of every candidate rule.
The candidate rules use the
Count action, not Block , so the request still reaches CloudFront. Instead of blocking, they inject two request headers: decoy-needed , and a miss directive of block or allow that tells the edge what to do when no decoy exists yet. WAF owns the policy decision; the edge owns readiness.2. A CloudFront Function substitutes the decoy in place
A viewer-request CloudFront Function checks a compact marker in CloudFront KeyValueStore — one small record per source page, never one route per decoy page. When a ready decoy exists, the function rewrites the request URI to the decoy's key in a private S3 corpus bucket and calls
cf.updateRequestOrigin() to point the request there. The crawler receives 200 at the URL it asked for.That single rewrite does two jobs. It swaps the content without changing the public URL, and it draws the cache boundary: viewer-request rewrites are applied before CloudFront computes the cache key, so decoys occupy their own cache entries while an ordinary visitor asking for the same URL still gets the real page. Substituting content without rewriting would cache a decoy against the real URL and serve it to people.
When no decoy exists yet, the function logs a generation signal and executes the WAF directive — return a block response, or pass the request through to the real origin unchanged. The crawler never waits for generation.
3. Amazon Bedrock generates the fiction asynchronously
The generation signal flows from CloudFront Function logs through a CloudWatch Logs subscription to a parser Lambda, then to an Amazon SQS FIFO queue. This stage is what makes the maze a mirror: the pipeline fetches the real page the same way a visitor would — including a headless renderer that executes the page's JavaScript, so the decoy reflects the hydrated page rather than an empty shell — validates that it got real content, and then generates an interlinked decoy corpus with Claude on Amazon Bedrock (Claude Opus 5, falling back to Claude Sonnet 5).
The result is published as an immutable version: content goes to a versioned S3 corpus, an immutable Amazon DynamoDB snapshot is the source of truth, and the KeyValueStore marker is promoted only after every object is verified. Partial publication can never become the active graph, and a failure leaves the previous version serving.
Source content is treated as untrusted model input — scripts, hidden content, and embedded instructions are stripped before anything reaches the model, and only deploy-time allowlisted hosts are ever fetched. A request-supplied host or arbitrary URL is never fetched.
4. Cost stays bounded and fiction stays fresh
Which paths get crawled is the adversary's choice, so an unbounded generation loop would be an attacker-controlled bill. The sample bounds it three ways:
- An admission budget. The parser admits at most
GEN_BUDGET_PER_WINDOWnew pages per window (default 20 per hour). Everything past the cap is dropped before the queue, with a metric. The crawler notices nothing — it keeps getting the WAF miss directive, exactly as it does while a decoy is being generated. The worst-case Bedrock bill is the cap times the window rate: chosen by you, not by the crawler.
- Failure backoff. Paths that can never produce a decoy — a login wall, an asset with no archetype — back off in steps from 5 minutes up to 24 hours, and the edge suppresses generation asks until the backoff passes.
- Rotation. A decoy that never changes while the real page evolves is its own fingerprint. Decoys past their time to live (default 24 hours) are still served — nobody is harmed by a slightly stale fake page — but a full regeneration runs in the background: new fiction, new page ids.
What you can observe afterwards
Attribution comes from the logs, not the artifact. Every serve logs a derived page id, the source page, the corpus version, the media type, and the user agent. Every link a decoy emits carries the id of the page that offered it, so the crawl graph rebuilds from the logs without depending on the
Referer header. Decoys also carry read-back beacons — an HTML decoy's own stylesheet, a JSON decoy's image value — so anything that actually renders or displays the content fires a signal without choosing to.Everything below is the stock AWS console, populated by one pass of the included test scripts against a deployed stack.
The Amazon CloudWatch dashboard is the hot view. The activity chart shows the test's crawl bursts, and the log tables answer the id-level questions in place: who read which decoy and through which client, and which real path asked for generation and why:

Further down the same dashboard, one row per decoy correlates served against read, and the arrivals tables catch clients coming back on invented URLs — including a browser arriving from an off-site referrer. That arrival is the signal the whole chain exists to produce: a person landing on a URL that only ever existed inside the fiction means a model surfaced it to them:

The hot view has a limitation: the serve-to-arrival chain can take months, and a CloudWatch log group deletes the serve record long before the arrival. So the durable view is Amazon Athena over an S3 log lake with 400 days of retention. The stack deploys the analysis as saved queries, ready to run in the console with no setup:

Running the per-decoy correlation against the lake returns the same one-row-per-decoy view over a window that outlives the log groups:

Deploy the sample
You need:
- An AWS account with the CDK bootstrapped in
us-east-1— the stack must live there, because CLOUDFRONT-scope AWS WAF, CloudFront KeyValueStore, and the CloudFront Function log group all require it.
- Amazon Bedrock model access to the Claude Opus 5 inference profile and the Claude Sonnet 5 fallback in that account.
- Docker running locally — the deploy builds two arm64 container images.
- Node.js 20 or later and npm.
Then:
cd infranpm installnpx cdk deployOne stack deploys everything: the WAF web ACL, the CloudFront distribution and function, the KeyValueStore, the generation pipeline, the dashboard, and the Athena lake. Verify it end to end with
./scripts/e2e-test.sh , which exercises the full scenario matrix — detection, in-place substitution, both miss directives, beacons, rotation, and the spend ceiling — against the live stack.One warning before you deploy: this sample bills while deployed. AWS WAF Bot Control is a paid managed rule group with a monthly subscription plus per-request charges, every new crawled path admits a Claude Opus 5 generation (capped, but not zero), and CloudFront, Lambda, S3, DynamoDB, and the 400-day log lake all accrue normal usage charges. Tear the stack down when you aren't actively using it:
cd infranpm run destroyTeardown removes every stack resource, including the S3 buckets and log groups. No manual cleanup is required.
What this design can and can't claim
I want to be honest about the boundaries, because they shaped the design:
- The maze is an economic deterrent, not a cloak. A crawler that can present two identities detects the maze by diffing the responses, and any AI-text detector doubles as a maze oracle because decoy prose is model-generated. The goal is to make unwanted crawling expensive and observable, not undetectable.
- A beacon proves rendering, not training. A fired beacon shows that something rendered or displayed the decoy. A bytes-only consumer fires nothing, and a model that reproduces the prose without the URL makes no request at all — that case is only findable by searching for the decoy's invented canary phrases in model output.
- Some pieces are design-only. The sample doesn't implement generation validators, trusted HTTP source adapters, or the automated feedback loop that promotes verified deep-traversal clients into expiring WAF IP sets. The evidence for that loop — every serve stamped with page id, version, and WAF verdict in the Athena lake — is deployed. The promotion step is documented as design.
And to repeat the note from the top: this is not a production offering. If you ever adapt the ideas beyond a sandbox, that path starts with your own legal and security reviews, then observation mode — classification and logging before any substitution — with one source origin and one decoy bucket, expanding only after the readiness, snapshot, and cache contracts are proven in your environment.
Try it
The repository at github.com/arditti/sample-cloudfront-waf-ai-agents-maze includes the full CDK app, the test scripts, a seeded sample site, and a README that goes much deeper: the publication invariants, the URL and cache contract, the attribution data flow, and the alternatives I rejected along the way (WAF custom responses, KeyValueStore as a per-page route database, inline Bedrock calls, and Lambda@Edge).
Deploy it, run the test scripts, and watch the dashboard fill in. The moment that made it click for me was seeing the same URL return two different realities depending on who asked — and knowing the logs could tell me exactly who got which one. Fluid UIs taught us that one page doesn't have to mean one rendering. This sample just asks who else that lesson applies to.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article