Amazon Bedrock Throttling: An Architecture for Surviving 429s
Your GenAI feature works beautifully in the demo. Then a launch, a batch job, or a viral moment lands — and Amazon Bedrock starts returning ThrottlingException (HTTP 429). Requests fail, latency spikes, and users see errors. Almost every Bedrock throttling incident I've seen traces back to the same root cause: teams treat quotas as an afterthought instead of a design input.
Throttling isn't a bug. It's Bedrock deliberately protecting shared capacity when you exceed the limits assigned to your account. The good news: it's completely designable-around. Here's the framework I use.
First, understand what's actually being limited
Bedrock enforces two quotas per model, concurrently:
- RPM (requests per minute) — how many InvokeModel/Converse calls you can send per minute.
- TPM (tokens per minute) — how many tokens (input and output, combined) you can process per minute.
You get throttled the instant you cross either one. This is the detail that trips people up: a workload with a few long-context RAG calls can blow the TPM ceiling while sitting nowhere near the RPM limit. A chatbot firing many tiny prompts does the opposite. You cannot reason about throttling without knowing which quota you're actually hitting — so instrument both before you tune anything.
The decision framework: four levers, in order
When you're getting 429s, reach for these in sequence — cheapest and fastest first.
1. Client-side retries with backoff (do this first, always). Transient throttling should never surface to a user. Configure the SDK to retry with exponential backoff and jitter. In boto3 that's one line:
python
import boto3
from botocore.config import Config
cfg = Config(
retries={"max_attempts": 5, "mode": "adaptive"}, # backoff + jitter, client-side rate limiting
read_timeout=120,
connect_timeout=10,
)
bedrock = boto3.client("bedrock-runtime", region_name="us-west-2", config=cfg)
adaptive mode is the key choice: unlike standard , it also throttles itself client-side when it detects sustained 429s, smoothing bursts instead of hammering the API. For anything it can't recover, wrap your own guard so a genuine failure degrades gracefully rather than 500-ing:
python
from botocore.exceptions import ClientError
def converse_resilient(bedrock, **kwargs):
try:
return bedrock.converse(**kwargs)
except ClientError as e:
if e.response["Error"]["Code"] == "ThrottlingException":
# queue for async retry, serve a cached answer, or return a friendly fallback
return {"fallback": True}
raise
2. Shape the load so you burn quota efficiently. Retries buy time; they don't add capacity. Reduce how much quota each request consumes:
- Trim tokens. Shorter system prompts and tighter retrieved context cut TPM directly.
- Turn on prompt caching for large, stable prefixes — cached tokens don't re-burn your TPM budget on every call.
- Right-size the model. Routing simple requests to a smaller, cheaper model frees headroom on your expensive one.
- Cap
maxTokens . Output tokens count against TPM too, so bound them.
3. Spread demand across capacity with cross-Region inference. A cross-Region inference profile lets Bedrock route your calls across multiple Regions, raising effective throughput and adding failover. (I've written a dedicated deep-dive on choosing a profile and the data-residency tradeoffs — Global vs geographic — so I won't rehash it here; just know it's the right lever after you've fixed retries and payload size, not before.)
4. Raise the quota — last, not first. Only after the above should you request an RPM/TPM increase through Service Quotas. Raising limits on an app that retries badly or sends bloated prompts just moves the failure to a higher, more expensive threshold. Fix efficiency first; buy capacity second.
A simple readiness checklist
Before any Bedrock feature ships to production, I want yes to all five:
- Are retries configured with
adaptive mode and a sane max_attempts ?
- Is there a graceful fallback for exhausted retries (cache, queue, or friendly message)?
- Am I emitting CloudWatch metrics/logs that separate RPM throttles from TPM throttles?
- Have I load-tested at realistic peak, not average?
- Do I know my current per-model quotas — and have I requested increases with lead time, not during the incident?
Why this matters / what I'd do
Throttling is where GenAI prototypes quietly die on the way to production. The teams that ship reliably treat quotas as a capacity-planning exercise, not an error to catch. My default posture: assume you will be throttled, make transient throttling invisible through retries and caching, keep per-request token cost low, and reserve quota increases for genuine sustained growth you've measured. That ordering keeps both your error rate and your bill under control.
The most common mistake? Requesting a quota bump as step one — it hides an inefficient design behind a bigger number until the next spike.
Your turn: when Bedrock last throttled you, was it RPM or TPM that gave out first — and did you know before you checked the logs? Share your war story below.
.
.
.
#AWS #Bedrock #GenAI #MachineLearning #CloudArchitecture #Reliability
Loading article