I Load-Tested My AI Gateway. 14% of the Messages Never Landed.
By Rajpreet Gill | AWS Community Builder – AI Engineering
Part 2 of "Building a Serverless AI Gateway on AWS." Part 1 designed the airport. This part puts real passengers through it.
Part 1 ended with a promise:
"It's serverless, so it scales" isn't evidence. Test the whole journey, not just Lambda.
So I built the whole airport WAF, Cognito, AppSync Events, Lambda, DynamoDB, Bedrock, Firehose, S3, Glue, Athena and CloudWatch. Then I pointed k6 at it and pressed go.
The very first test failed.
Nothing crashed. Every Lambda invocation returned 200. AppSync reported zero dropped events. Every dashboard was green.
And yet passengers were standing at the gate, waiting for a plane that had already landed.
This is the story of how I tracked that down using the tracking numbers from Part 1, the three surprises I found along the way and the changes that took message loss from 14% to zero.
The Short Version
| Before | After | |
|---|---|---|
| Streamed events lost | 13.8% | 0% |
| Requests that never finished | 9 of 100 | 0 of 100 |
| Messages received in the quota test | 37,445 | 1,617 (23× fewer) |
| Median time to first token | not measurable yet | 473 ms |
| Quota requests that slipped past the limit | not tested yet | 0 |
The gateway logic was right from the start. The part that broke was the one I'd dismissed as plumbing: getting the answer back to the passenger.
The Flight Plan
It's one CDK stack.
cdk deploy builds the airport, and cdk destroy tears it down.Testing the airport, not the aircraft
If a real model is slow, that's the model's problem. I wanted to know whether the gateway was slow. So most tests use a mock model: a fake Bedrock that streams at a steady, known pace.
1
Mock model: ~250 ms to first token, then 8 ms per tokenAnything slower than that is overhead the gateway added.
The passengers
| Tenant | App | Rules |
|---|---|---|
| Contoso | Support Assistant, Developer Assistant | Large quota, primary model with a fallback |
| Contoso | Knowledge Bot | Cheap models only (remember "who approved the expensive model?") |
| Fabrikam | Document Summarizer | A tiny 20,000 tokens/hour quota, built to be squeezed |
The test client
The k6 script behaves like a real chat app:
1
connect ──► subscribe ──► publish request ──► collect tokens ──► doneIt measures time to first token (TTFT) and total latency. It also measures one number I didn't know I'd need until I needed it: how many published events actually arrived.
Test 1: The Plane Landed. The Passenger Didn't.
Ten users, ten requests each. It's the most boring test there is, and it's the one that's supposed to pass.
| Result | |
|---|---|
| Successful requests | 91 of 100 |
| Answers that arrived incomplete | 62 |
| Requests that never finished | 9 (waited 60 seconds for nothing) |
My first thought was that the gateway was broken.
So I did what Part 1 told me to do at Gate 9 and pulled up one of the failed requests by its tracking number:
1
2
3
4
5
request_id: 3334c61c-89c3-4dd9-9dfc-4a4de4dec269
received tenant=contoso app=support-assistant
quota_reserved estimate=314 used=44810
completed status=200 model=mock-a ttft_ms=210 latency_ms=3219The gate agent had done everything right. The request was authorized and metered, it streamed 300 tokens, and it landed in 3.2 seconds with status 200. Every publish to AppSync came back successful.
Meanwhile the client, still connected, waited a full minute and heard nothing.
This is why the tracking number exists. Without that request ID, I'd have been staring at an average-latency graph and guessing.
The Investigation: Lost Baggage at AppSync
AppSync's own metrics made things more confusing, not less:
| Metric | Value |
|---|---|
| BroadcastDroppedEvents | 0 |
| FailedEvents | 0 |
| Events delivered to clients vs. events received | about 14% fewer went out than came in |
Nothing was reported as dropped, yet messages were disappearing.
So I added delivery accounting to the test. Every chunk carries a sequence number, the final message carries the total count, and the client compares what the gateway published with what arrived.
Clue #1: the bags come off the carousel out of order
Sometimes the final
done message overtook the last few chunks of text. My client treated done as "that's everything" and hung up while the rest of the answer was still in the air.AppSync Events doesn't guarantee delivery order, so the client has to put the pieces back in sequence itself.
Clue #2: some bags never come off the carousel at all
With ordering fixed, the counter still showed 257 of 1,858 events lost (13.8%). These weren't late. They were gone.
The root cause: a new gate for every flight
My client did what most AppSync examples show. For every request, it:
- opened a new WebSocket,
- subscribed to a new channel,
- waited for
subscribe_success, - and then published the request.
subscribe_success means AppSync has accepted your subscription. My working theory is that it doesn't mean every node that might broadcast your answer already knows about it yet.The fastest requests gave it away. Cache hits, which answer in milliseconds, sometimes lost everything. The gate announcement went out before the gate was really open.
The Fix: One Gate per Passenger
Real travellers don't get a new boarding pass every time they buy a coffee. A chat app shouldn't open a new subscription for every message either.
1. One long-lived connection, one subscription per session
1
2
3
4
BEFORE every request ──► new socket ──► subscribe /responses/{user}/{request}
AFTER once per session ──► subscribe /responses/{user}/{session}/*
every request reuses it ──► answer lands on /responses/{user}/{session}/{request}An AppSync JavaScript handler checks that
{user} in the channel matches the caller's Cognito identity, so nobody can subscribe to someone else's answers.2. Sequence numbers on every chunk
The client reassembles the answer in order, whatever order the chunks arrive in.
3. done carries the complete answer
If a chunk goes missing, the final message repairs it.
4. Terminal messages are sent twice
Losing one streaming chunk is annoying. Losing the last message leaves a user staring at a spinner forever.
1
2
3
4
5
def done(self, summary):
self.flush()
event = {"type": "done", "chunks": self.seq, "text": self.full_text, **summary}
for copy in (1, 2): # delivery is best effort
self._send({**event, "copy": copy})The result: same test, same 100 requests
| Version | Events lost | Incomplete answers | Failed requests |
|---|---|---|---|
| v1: new subscription per request | not measured yet | 62 | 9 |
| v2: + sequence numbers, done carries the full text | 257 of 1,858 (13.8%) | 0 (repaired by done) | 4 |
| v3: + one connection per session | 0 of 1,796 | 0 | 0 |
Now the Latency Numbers Mean Something
Once delivery could be trusted, latency could be trusted too:
| Metric | Median | p95 | p99 |
|---|---|---|---|
| Time to first token | 473 ms | 2.2 s | 2.5 s |
| Total latency (300-token answer) | 3.4 s | 4.8 s | 5.4 s |
The median: about a quarter-second gateway tax
Inside Lambda, the first token was ready after a median of 249 ms, and nearly all of that was the mock model's own "thinking" time. The client saw it after 473 ms.
1
473 ms (client) − 249 ms (inside Lambda) ≈ 224 msSo the full trip from client to AppSync to Lambda and back costs roughly a quarter of a second. That's a fair price for a serverless WebSocket you never have to patch, scale or babysit.
The p95: ten cold starts
The p95 tells a different story. The Lambda logs show exactly 10 cold starts in that run: ten brand-new users arrived at the same moment, so ten execution environments started at the same moment. Those first requests make up the tail.
This is exactly why Part 1 said averages hide the users who are suffering. The median says "fast". The p95 says "unless you're first in line".
Test 2: The Baggage Scale Under Pressure
Next, Fabrikam and its tiny 20,000 tokens/hour quota. The goal was to hammer it and check that the baggage scale from Gate 5 holds.
This test turned up three more surprises before it produced a clean result.
Surprise #1: everyone waiting at the same gate
My first session fix subscribed each session to
/responses/{user}/*.In the quota test, 20 virtual users all signed in as the same Fabrikam user. So every session received every other session's answers and quietly threw them away.
| Channel design | WebSocket messages received |
|---|---|
| /responses/{user}/* | 37,445 |
| /responses/{user}/{session}/* | 1,617 |
That's the same test with 23× fewer messages.
AppSync Events bills per message, so this bug hurts your bill as well as your bandwidth. And it grows quadratically: if one user has N open sessions, every answer is delivered N times. A user with a laptop, a phone and three browser tabs is paying for everyone's traffic five times over.
Give every session its own channel.
Surprise #:2 the runway only had ten slots
That same 20-user run was also strangely slow, with a p90 TTFT of around 7 seconds. The gate agent's metrics explained it:
| Metric | 20 users | 8 users |
|---|---|---|
| Lambda throttles | 71 | 0 |
| Max concurrent executions | 10 | 8 |
| Oldest request waiting in the queue | 30.8 s | ~0.1 s |
My new AWS account had a Lambda concurrency limit of 10.
AppSync invokes the gate agent asynchronously, and asynchronous invocations don't fail when you're over the limit. They queue and retry. Nothing was dropped and every request eventually ran, some of them 30 seconds later.
This is the most dangerous kind of failure: no errors, just waiting. An error-rate dashboard would have shown a perfectly healthy system.
Check your account's Lambda concurrency limit before you load-test, and request an increase well before launch day. And put concurrency on your control tower dashboard, not just error rates.
The clean run: 8 users, 1 minute
With the channel fix in place and traffic kept under the concurrency limit:
| Result | |
|---|---|
| Requests that boarded | 40 |
| Clean 429 Token quota exceeded | 195 |
| TTFT on the ones that boarded | median 504 ms, p95 582 ms |
The DynamoDB conditional write did exactly what Part 1 promised: requests arriving at the same moment could never both slip under the limit. Fabrikam hit its ceiling, and every request after that got a clean, immediate "overweight baggage" response instead of a surprise on next month's invoice.
Surprise #3: the last 1.6%
I'd made
done send twice but forgot that a 429 is a terminal message too. So I made every terminal message (done, error and 429) go out twice, and ran the quota test again.The rerun got better and worse at the same time.
| Result | |
|---|---|
| Clean 429 Token quota exceeded | 287 |
| Requests that got past the limit | 0 |
| Requests that never heard back | 5 of 315 (1.6%) |
Enforcement was perfect. Not one request slipped past the limit.
But five requests still got nothing: not one copy of their 429, not a single word. Sending the message twice didn't change them at all, so my theory was wrong. Something else is losing those requests, either on the way in, before they reach Lambda, or on the way back out.
Part 1 promised "clean 429s, every single time." That's true for the quota logic. It isn't yet true for delivery
I've added a new metric that records whether AppSync accepted each request, so the next run will show which way they're going missing.
What I'd Tell You Before You Build Yours
- Streaming over pub/sub is best effort. Design for it. Use sequence numbers, a final message that carries the whole answer, and terminal messages sent more than once.
- Don't open a subscription per request. Hold one connection per session and route answers by request ID.
- Give every session its own channel, or your users will pay to receive each other's traffic.
- Measure delivery, not just latency. "Published" and "received" are two different numbers. Count both.
- Async invocations hide throttling as latency. Watch concurrency and event age, not just error rates.
- Keep the tracking number. Every mystery in this post was solved by pulling up one request ID.
Final Boarding Call
In Part 1, the gateway was a diagram. In Part 2, it met real traffic, and the traffic had opinions.
The core logic was right the whole time. Every request was authorized, metered, routed and recorded. What broke was the part I'd assumed was just plumbing: getting the answer back to the passenger.
The LLM is the aircraft.
The gateway is the airport.
The gateway is the airport.
And a flight isn't finished when it lands. It's finished when the passenger walks out of the terminal with their bags.
Sources:
- Build a serverless AI Gateway architecture with AWS AppSync Events
- Understanding channel namespaces
- AppSync event handlers
- Invoking a Lambda function asynchronously
- Understanding Lambda function scaling
- Lambda quotas
- Grafana k6 WebSockets
Code for this post: https://github.com/Rajpreet12/aws-serverless-ai-gateway
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article