Building a Resilient Inference Engine is Essential for GenAI Workloads
Managing Traffic Spikes with Amazon Bedrock — Part 2
Written by Ryan Sachs Startup Solutions Architect, Nick McCord Senior Startup Solutions Architect, Alex Moening Senior Edge Solutions Architect
Introduction
In Part 1 , we explained why traditional retry patterns fail for Amazon Bedrock workloads and introduced proactive rate limiting as the alternative. At that time, we hypothesized that exponential backoff with jitter creates feedback loops which amplify failures. In this post, we put the theory to the test.
Since Part 1, Amazon Bedrock's quota model has evolved. More and more models are standardizing on enforcing tokens per minute (TPM) instead of both TPM and requests per minute (RPM) limit:
- Nova 2 Lite on
bedrock-runtime: 8M TPM (combined input + output). 2000 RPM.
- Claude Opus 4.8 on
bedrock-runtime: 30M TPM (combined input + output). No RPM.
- Claude Sonnet 5 on
bedrock-mantle: 3M input TPM / 300k oTPM. No RPM.
- GPT 5.6 on
bedrock-mantle: 10-20M input TPM / 1-2M output TPM. No RPM.
This article focuses on TPM-based throttling and how to design your inference backend for high availability with more consistent and predictable throughput. The core tradeoff is introducing latency into your workload.
How Workloads Fail Today
Across industry verticals and spend profiles, customer inference workloads tend to be consistently inconsistent. One minute you get 3x what your quota can handle and then you might spend the next 20 minutes utilizing a small percentage of your throughput. Traffic shaping smooths out your traffic so your workloads consistently utilize the available throughput. We've seen first hand from many customers workloads similar to this profile where traffic shaping can have a meaningful impact on inference resiliency:
Putting traffic shaping in front of the inference workload would change the throughput to something like the image below without requiring any quota increases.
These few traffic spikes result in poor user experiences and frustration that can lead to loss of business. The pain of downtime and/or increased error rates for limited timeframes also leads to undesirable administrative overhead that can be reduced and in some instances eliminated, through traffic shaping.
How It Works
The traffic shaper reserves 85% of your quota capacity for constant throughput leaving 15% of your capacity as a margin of safety. Requests are drained from the queue with a siding window and regeneration-aware pacing which together normalize constant throughput with minimal throttling. We found in load testing that increasing beyond 85% can begin to introduce minor throttling while decreasing below 85% leaves throughput on the table. This configuration can be altered per model and per workload.
The traffic shaper intentionally overrides default
boto3 settings to remove retries. Only one request is sent to Bedrock per request coming in. This is intentional to ensure the highest probability of a successful request. Customers who are implementing this have event driven experiences or make UX tweaks to make it clear that the request is in a queue.What does this look like in practice? If you have a quota of 1 million tokens per minute, this solution will attempt to utilize 850k TPM constantly. When your application receives a traffic spike of 3 million tokens in 15 seconds, the queue will absorb the spike and will drain in ~3.5 mins, which in many cases will be preferred compared to the application experiencing down time or a high percentage of failed requests.
Architecture
The implementation uses AWS serverless services:
- DynamoDB for request tracking, queue management, and model configuration (single-table design)
- Step Functions for request lifecycle orchestration using the callback pattern (
waitForTaskToken)
- Budget Manager Lambda to add requests to the queue
- Queue Processor Lambda to drain queued requests as capacity regenerates
- Bedrock Processor Lambda to call the Bedrock API for queued requests
Know Your Traffic Pattern
Understanding your traffic pattern determines the right strategy. In some workloads, the latency will not be tolerable and sometimes the answer is to increase the quota or to utilize multiple inference backends. In many scenarios, traffic shaping can give you resiliency and the capability to know when your request is unlikely to be fulfilled.
Should you use Retry/Jitter?
We fired 1,000 requests at Amazon Nova 2 Lite in a single burst within 30 seconds. Each request carried ~18,500 tokens, producing 18.5M tokens against a 4M TPM quota: a 4.6× overload.
Metric | No Retry | Retry + Jitter | Traffic Shaper |
|---|---|---|---|
Success rate | ❌ 74.4% | ❌ 14.3% | ✅ 100% |
Failed requests | 256 (lost) | 857 (lost) | 0 |
Total API calls | 1,000 (1×) | ~3,900 (3.9×) | 1,000 (1×) |
p99 latency | ~0.3s (fast fail) | ~29s | ~260s (queue drain) |
Our retry implementation follows the industry-standard pattern recommended by AWS SDKs and distributed systems best practices:
- Max retries: 3 attempts after the initial request (4 total)
- Backoff formula:
min(30s, 1s × 2^attempt)— producing delays of 1s, 2s, and 4s
- Jitter: Full jitter —
random.uniform(0, computed_delay)per attempt
- Retry condition: Only on HTTP 429 (ThrottlingException)
In our 1,000-request test: 256 initial throttles triggered retries, which consumed quota needed by other in-flight requests, causing more throttles, triggering more retries. After 3 retry rounds, 1,000 original requests had generated 3,900 total API calls. Of those, 2,568 accomplished nothing except stealing capacity from legitimate first-attempt requests. This is why "no retry" (74.4%) outperformed "retry + jitter" (14.3%).
For production inference workloads implementing retries is actually counter productive and we do not recommend it for spiky workloads.
Traffic Shaping at Scale
The Nova 2 Lite test proved the concept on a single model demonstrating the negative impact of using retries and jitter. To validate production readiness, we ran extensive testing across Bedrock Mantle and Bedrock Runtime endpoints. We re-ran the validation campaign against Claude Sonnet 5 and GPT-5.6 Luna at roughly 3× their configured quotas, comparing direct-to-Bedrock behavior against the shaper. We tested breaching Sonnets iTPM and GPT Luna's oTPM quotas demonstrating that the solution can fulfill either rate limiting quota.
Key Findings
- The shaper delivers more token throughput and zero throttles at identical offered load. mean delivered tokens were higher through the shaper than direct-to-Bedrock. This was valid on the dimension each model's quota actually binds (input tokens for Sonnet 5, output tokens for Luna).
- Claude Sonnet 5 (input tokens): 9,972,589 (shaper) vs 6,626,781 (direct) — +50.5%
- GPT-5.6 Luna (output tokens): 7,032,005 (shaper) vs 4,214,007 (direct) — +66.9%
- Total per minute token throughput achieved is dependent on model latency. Queue drain time is predictable.
- Claude Sonnet 5: 2,541,564 measured vs a 2.55M/min iTPM allocation. Completion time averaged 235.4s to drain (spread 12.6s across 5 runs),
- GPT 5.6 Luna: 1.2m oTPM measured vs 1.7 oTPM allocation. Completion time average 326.9s (spread 6.6s across 5 runs).
- Core tradeoff of throttling vs latency is the key decision
- Mean completion was 235s (Sonnet 5) and 327s (Luna)
- The tradeoff in these tests would be a 4-5.5× longer window in exchange for eliminating a 37–41% loss rate.
- Adding retry and jitter does not improve overall throughput. The same average throughput is accomplished without it while retries double the likelihood of an individual request failing.
Testing Methodology
Achieving the request throughput necessary required using the distributed load tester on AWS . All requests sent to the models did not use prompt caching and were of similar size. For production workloads, both of these factors will impact model latency and you may want to tweak the way you drain items from the queue accordingly.
Scaling Bottlenecks
At scale, the ceilings are AWS service limits with well-understood solutions -- categorically different from a Bedrock throttle, which drops your request with no recourse. Here are a few of the scaling challenges we ran into:
Service | Limit Hit | Default Quota | Mitigation |
|---|---|---|---|
Step Functions | StartExecution API throttle | 1,500/s refill, 5,000 bucket | Request quota increase |
Lambda | Concurrent executions | 1,000 (default) | Request increase; reserved concurrency |
DynamoDB | Single-partition read/write throughput | 3000 RCU (Read Capacity Units) | Partition sharding. |
DynamoDB | Single-partition write throughput | 1,000 WCU (Write Capacity Untits) | Partition sharding. |
Bedrock | Model latencies | N/A | Update queue draining algorithm |
We found when load testing Opus that 400 RPS caused DynamoDB to throttle through read capacity units. For every two or three hundred RPS, we would recommend configuring another shard for production workloads. The pattern for how to solve this is documented here and is a future enhancement to this project.
We increased our step function start execution limit and lambda concurrency to 5000 each and did not run into any further throttling in our load tests from those services.
GPT-5.6 Luna, in particular, experienced longer per-request latencies during load testing, which limited the TPM achieved under the current implementation. For production workloads you may need to customize the speed at which you drain your queue to fully utilize your allocated quota on Bedrock.
Conclusion
Inference workloads are inherently spiky and must operate within throughput constraints. Increasing quota limits can provide short-term relief, but quota increases alone do not add resiliency, predictability, or high availability to your product. Traffic shaping, by contrast, addresses outages more permanently by buffering consistent traffic to Bedrock within your limits. This approach is especially valuable if you occasionally spike above your quota or if you can tolerate the latency tradeoff of request queuing.
Exponential backoff doesn't rescue a throttled workload — it deepens the hole. Load tests with Nova 2 Lite and Sonnet 5 both demonstrate your success rate per request plummets. The reflex that's supposed to add resilience is the thing costing you the most. Traffic shaping flips the model. Instead of firing into a wall and retrying the wreckage, you pace into the quota and let a queue absorb the spikes. In our tests that meant near-zero dropped requests and near-zero throttling, at the cost of latency you can measure, predict, and design around. For a spiky workload, that beats a quota increase and a retry loop every time.
What to do with your inference workload:
- Profile your traffic. Find your baseline and your spikes. If you look like Scenario A or B, shaping likely solves your throttling outright. If you're C or D, shaping still helps — pair it with more quota or a second backend.
- Kill your retry-on-429 logic for spiky Bedrock inference. It's not helping. The data says it's hurting.
- Pace your drain to 70–85% of quota and watch your error rates and token throughput.
We're actively working to open-source this solution. In the meantime, reach out to your AWS account team for a consultation on whether this pattern fits your workload. We strongly recommend exploring this solution because it solves the underlying problem rather than simply raising limits.
Appendix
Per-Arm Token / TPM Summary — Claude Sonnet 5, GPT-5.6 Luna (5 runs per arm)
Claude Sonnet 5 — Retry + Jitter (3 runs)
Amazon Nova Lite (8M TPM quota — 30 RPS for 3 mins, 1.4x TPM)
About the Authors
Ryan Sachs is a Startup Solutions Architect at AWS specializing in production GenAI systems and inference infrastructure. He works with early-stage founders and scaling companies across to design resilient, cost-optimized AI workloads on AWS. Outside of work, Ryan is a passionate NFL and golf enthusiast.
Nick McCord is a Senior Startup Solutions Architect at AWS focused, also specializing in Edge Service and Perimeter Protection. He is based on Chicago, loves his three rescue dogs, and riding motorcycles.
Alex Moening is a Senior Edge Solutions Architect at AWS who brings innovative thinking and deep problem-solving expertise to cloud technology. He specializes in Content Delivery Networks and has worked on numerous high-scale applications for his customers, including some of the largest names in the delivery ecosystem. Alex actively contributes to industry research, including the annual Web Almanac, and shares his expertise through interactive sessions and presentations at industry events such as AWS re:Invent.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article