AWS Builder Center

Load test your Amazon Bedrock model shortlist before production

Learn how to compare Amazon Bedrock models under burst and sustained load, distinguish quota, capacity, service, and client failures, and avoid misleading performance conclusions.

AWS Solutions Architect
John Walsh, Solutions Architect, AWS | Harshvardhan Chunawala, Solutions Architect, AWS | Tiago Costa Tomas, Sr Bedrock Specialist Solutions Architect, AWS
You have probably evaluated your foundation model shortlist on cost, latency, and accuracy. There is a fourth dimension that standard evaluation frameworks do not cover: how a model behaves under production-scale concurrency. It is not uniform across models, and if you do not test for it, you will discover the differences in production, which is the most expensive place to find out.
We learned this by systematically load-testing a set of models on Amazon Web Services (AWS) using  Amazon Bedrock , across burst, sustained, and service-tier scenarios. Two similarly priced models, given near-identical bursts of concurrent requests, behaved very differently. One completed most of the burst. The other rejected most of it with capacity errors. Nothing in a quality eval or a latency benchmark would have predicted that.
This article shows you how to run the same evaluation on your own shortlist, and just as important, how to correctly read what comes back. When a load test produces errors, 429 responses, 503 responses, service errors, and client timeouts have different causes and require different responses.
A quick scope note: serving capacity, quotas, and pricing all change over time and vary by account and workload. Treat the diagnostic principles described here as durable. Treat any specific behavior you observe as a snapshot, not a benchmark of any model or of Amazon Bedrock generally.

Know what you are actually measuring

When you push load at any model endpoint, everything that is not a 200 looks like "errors" on your dashboard. Before you draw any conclusions, learn to tell these apart:
What you see
What it actually is
What to do about it
Fast 429s, often within tens of milliseconds, with messages referencing throttling or an exceeded allowance
The request was rejected by a quota, allowance, or internal rate limit
Reduce the submission rate. If the applicable quota is exposed through Service Quotas, request an increase
503s with messages about exceeding on-demand capacity and ramping your workload gradually
Available throughput has not scaled to the requested rate, or Regional demand is high
Reduce the rate and ramp traffic gradually using the
 documented ramp-up procedure 
. For latency-sensitive traffic, test the Priority tier if the model supports it
500s, slow responses, client-side timeouts
Different conditions that must be separated. A 500 is a service error, while a client timeout means the client stopped waiting before receiving a response
Record HTTP errors and client timeouts separately. Set the client timeout based on the expected output size and concurrency
Misdiagnosing one as another wastes your time. A 429 tells you that the request was rate limited, but it does not tell you how the model would behave if admitted. The exact limit and available remediation depend on the endpoint, model, Region, and account. Check the error body and your applicable quotas before comparing models.
This leads to a corollary that is simple to miss: when 429 responses dominate a run, you are measuring a quota or rate limit rather than the model's serving behavior. In our testing, several models plateaued at effectively the same successful request rate, with the excess rejected as fast 429s. Resolve that limiting factor before using the run as a model comparison.
It helps to understand why the two Amazon Bedrock endpoints behave differently here. The  bedrock-runtime  endpoint manages fair-share access through per-account quotas. The  bedrock-mantle  endpoint uses advanced scheduling and work-queueing to deliver fair-share distribution while supporting higher initial throughput, so in most cases requests are served immediately, and in some cases a request is briefly queued while in-flight work completes. Reading that behavior correctly is the difference between a useful comparison and a misleading one.
One caution carries straight into production: how you retry matters as much as whether you retry. For occasional 503 responses, the  Amazon Bedrock scaling guide  describes bounded retries with exponential backoff and random jitter. Each retry waits longer than the previous one, while jitter prevents many clients from retrying at the same moment. Sustained 503 responses or repeated 429 responses require a lower submission rate because retries alone can add load while capacity is already under pressure. Your measurement client should not retry, so each original request and response is counted once.

What load testing revealed

Serving behavior varies materially across models under near-identical load. In our test cohort, success rates for the same burst varied widely on the standard service tier. Some runs completed in seconds; others ran to the full client timeout. That is not a knock on any particular model, since capacity allocation differs per model and changes over time. The takeaway is simpler: your model selection process should include a load test on your actual workload.
The priority service tier helped unevenly. Amazon Bedrock offers  service tiers , including a priority tier at a pricing premium. Comparing the same burst across tiers, some models improved dramatically. One went from partial success spread over a full minute to complete success in seconds. Others saw little change, and one did worse in our single-shot comparison, which likely says more about capacity state at test time than about the tier. Treat any tier delta you measure the same way: as directional until you have repeated it under matched conditions. The practical guidance is to test the priority tier on your actual workload before paying for it. For some workloads it is transformative; for others it changes little.

Run it yourself

You do not need heavyweight tooling. Any load generator that can authenticate requests and report per-status-code results works. We used a small custom Python CLI, but the methodology is what matters.
We tested models through the Responses API on the  bedrock-mantle endpoint  using AWS credentials. API support varies by model, so check whether each model supports Responses, Chat Completions, or another API before building your test payloads.
Start with one request, then scale deliberately:
  1. Smoke test. Send one request with a trivial prompt such as  "Reply with exactly: hi"  and cap the output to a few tokens. Use  max_output_tokens  with Responses. If you test another API, use the output limit parameter documented for that API and model. For reasoning models, follow the model documentation to minimize or disable reasoning where supported. Otherwise, reasoning may consume a small output budget before the model returns visible content. Verify your authentication, model ID, API support, and response shape before generating load.
  1. Burst. Send a group of simultaneous requests with a client timeout. This surfaces admission behavior under a sudden increase in traffic. Start with tens of requests and step up gradually.
  1. Sustained. Hold a fixed request rate for several minutes and dispatch requests open loop, meaning the tool sends requests on schedule whether or not earlier ones have returned. A closed loop tool waits for a response before sending the next request, so it slows down with the service and can hide the behavior you are trying to measure.
  1. Priority tier comparison. Repeat the same test on the Standard and Priority tiers. Standard is used when you omit  service_tier . To request Priority, add  "service_tier": "priority"  to the request body. Confirm the resolved tier in the response because model support varies and the resolved tier is the one that served the request. Other tiers are outside the scope of this comparison.
Whatever tool you use, make sure it records the status code, latency, and error body for every request, and breaks latency out per status code. The error bodies help you separate these outcomes.
Before you run anything at scale: load tests create real cost, consume available throughput, and exercise shared capacity. Only test in accounts you are authorized to use, review the  scaling and throughput guidance , and put a timeout and request cap on every run.

Read your results honestly

Load test results are simple to misread. These disciplines will keep your conclusions honest:
  • Batch elapsed time is not mean latency. If your workload is batch-oriented, you care about end-to-end completion of the whole batch. A run can show a great p50 and still take the full timeout window to finish, because the tail gates the batch.
  • Fast batches can be bad news. A batch that "completes" in 5 seconds may have completed because half the requests were rejected quickly. Always read success rate and elapsed time together.
  • Verify your client can actually generate the load. Check your tool's real dispatch rate against the target. If the client was the bottleneck, the run is invalid as a service measurement. You benchmarked your load generator.
  • Control for capacity state. Because available throughput scales with your recent traffic, runs made under different preceding traffic conditions are not directly comparable. Treat single shot comparisons as directional, and repeat them under matched conditions before deciding anything.
  • Do not retry in your measurement client. Each 429, 503, and timeout should appear in your results exactly once, so error rates reflect the service's raw responses rather than outcomes masked by retries. Many SDKs retry automatically, so explicitly set the retry count to zero and verify the behavior of your client library.
  • Size your client timeout to the workload. Under heavy concurrency, a large requested output can legitimately take longer than a 60 second read timeout. If your timeout is too short for the payload, you record a client timeout while the service may still be processing the request, which inflates your measured error rate.
  • Trivial payloads validate serving behavior, not latency. A short smoke test tells you about authentication, admission behavior, and errors. It tells you nothing about latency on your real prompts. Rerun with representative request bodies before quoting latency numbers.

The takeaway

Cost, latency, and accuracy are necessary but not sufficient. Before you commit to a model for a production workload:
  1. Load test your actual shortlist at your actual target throughput, with representative payloads.
  1. Distinguish quota and rate limiting, capacity errors, service errors, and client timeouts before comparing models.
  1. Resolve client limitations and dominant rate limits before drawing conclusions from the results.
  1. If you are considering the Priority tier, measure it on your workload instead of assuming the premium buys you headroom.
An afternoon of deliberate load testing is a lot cheaper than discovering serving behavior differences in production. Try it on your own shortlist. Start with one request, and see what your models do when you turn up the dial.
To go deeper, see  Scaling and throughput best practices ,  Service tiers for optimizing performance and cost , and  Endpoints supported by Amazon Bedrock  in the Amazon Bedrock documentation. If you have questions or want to share what your own load tests turned up, leave a comment.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article