AWS Builder Center

Absolute Latency Reduction: Measuring Amazon Bedrock Latency from India

A practical latency lab for Amazon Bedrock that measures how streaming, prompt caching, exact-response caching, model choice, and AWS Region affect time to first token and completion speed for users in India.

I Built a Latency Lab for Amazon Bedrock Because "Just Use Streaming" Isn't an Answer

Every AI product advice list says the same three things: stream the output, cache the prompt, pick a closer Region. Nobody says by how much, or whether the improvement is even real for your users.
I wanted a real answer for one specific case: a user in India calling Amazon Bedrock. So instead of shipping another chatbot wrapper, I built the thing I actually needed — a tool that measures which latency optimizations are worth doing, and which are placebo.

The real problem

AI apps feel slow for reasons that get bundled into one vague word: "latency." In reality it splits into pieces that behave completely differently:
  • Time before the first word appears (TTFT)
  • Time until the full answer finishes
  • How fast tokens stream once they start
  • Whether a cache silently made the whole thing instant
  • Whether the Region you picked helps at all from an Indian network
Every "optimize your AI app" post picks one of these and calls it the fix. None of them prove it changes the number that matters. I built Absolute Latency Reduction to stop guessing.

What I built

Two pieces, both intentionally small:
 relay/  — A streaming gateway on an AWS Lambda Function URL (Node.js 22), sitting in front of Amazon Bedrock's  ConverseStream . Every request can flip exactly one switch: streaming on/off, Bedrock prompt caching on/off, an exact-answer cache in DynamoDB on/off, which model, and which Region.
 bench/  — The harness that actually runs the experiment: fires every configuration as its own randomized arm, then reports TTFT, completion time, and tokens/sec as p50/p95/p99 with paired bootstrap confidence intervals.
The relay also serves its own demo page directly at the Function URL, so there's no CloudFront or S3 bucket needed to show it working — one Lambda, one URL, done.

Built on AWS

Piece
AWS service
Job
Gateway
Lambda Function URL
Fronts Bedrock, applies the lever under test
Model calls
Amazon Bedrock (ConverseStream)
The actual inference
Exact-match cache
DynamoDB
Skips regeneration for repeated prompts
Multi-region test
Separate relay per Region (ap-south-1, eu-north-1, us-east-1)
Isolates network + Region as one variable
No CloudFront, no extra infra — deliberately, so the architecture doesn't add its own latency to the thing being measured.

What fought back

The build itself wasn't the hard part. Making the comparison trustworthy was.
Accidental caching. Early on, control requests could silently hit the same cache as the experiment, making prompt caching look better than it was. Fix: every request gets a random salt prepended to the system prompt, so nothing shares a prefix unless it's the arm actually being tested for caching.
Retries hiding failures. A naive client retries on throttling, which quietly inflates or deflates latency depending on when it happens. I made the Bedrock client single-attempt only — a throttle shows up as a recorded failure, not a smoothed-over number.
Fake precision. It's easy to print a p99 from 12 samples and have it look authoritative. I hard-gated it: no p95 under 20 samples, no p99 under 100. If there isn't enough data, the report leaves it blank instead of lying.
Unequal outputs. Comparing completion time is meaningless if one arm generated twice as much text. The harness checks output length and flags the comparison as invalid when it drifts, pointing to tokens/sec instead.
The account itself fought back. This is the part I'm not going to dress up: my AWS account is still in new-account verification, so Bedrock model calls currently return  NOT_AUTHORIZED  for every model, including Amazon's own. No live Region can be tested until that clears.

Execution: what actually works today

Rather than fake a result, I built the entire system against a synthetic mock model and proved every non-Bedrock-dependent part is correct:
  • 43 automated tests covering the relay, the statistics, the end-to-end harness, and the page logic
  • A working local relay and live demo page, with a hard-coded red SYNTHETIC banner whenever it's not talking to real Bedrock
  • A publish script ( publish-results.js ) that refuses to push any run containing mock-model data to the public results page
That refusal is the point. A benchmarking tool that can't tell the difference between fake and real numbers isn't a benchmarking tool — it's a chart generator. Mine can, and it says so on the page.

What I learned

Four things I didn't know going in:
  1. TTFT has to be measured at the client, on the first real token — not the first byte, not a "typing" placeholder. Those two definitions can differ by hundreds of milliseconds.
  1. A cache can contaminate your control group without anyone noticing, and the fix (unique prompt salt) is one line of code most benchmarks skip.
  1. Region choice is an empirical question, not an assumption — "closer" doesn't automatically mean faster once you account for where Bedrock actually runs a given model.
  1. The most valuable thing I shipped wasn't a speed-up — it was a way to tell a real speed-up from a coincidence.

What's next

The moment Bedrock access clears, the plan is mechanical: deploy one relay per Region, point the harness at all three from a fixed Bengaluru vantage point, run 10+ rounds, and publish only the runs that come back without the SYNTHETIC banner. You can check the account block yourself with one command, no tokens spent:
aws bedrock get-foundation-model-availability --model-id amazon.nova-lite-v1:0 --region ap-south-1 --profile hackathon

Why this matters beyond the hackathon

Every team building an AI product in India is going to hit this exact question: is my app slow because of the model, the Region, or something I can actually fix? Most will guess. Absolute Latency Reduction is a small, reusable way to stop guessing — built on Lambda, Bedrock, and DynamoDB, and honest about the one number it can't show yet.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article