Build Production-Scale AI Inference Systems on Amazon Bedrock
A practical guide to capacity planning and inference optimization on Amazon Bedrock.
If you're building generative artificial intelligence (AI) applications using managed AI model APIs, you've likely chosen this path for faster time-to-market and reduced operational overhead. To get the most from this approach at production scale, you need to understand the capacity model and inference options these managed APIs provide — so you can select configurations that match your workload's performance and cost requirements. Rather than simply requesting more quota as demand grows, a structured approach to capacity planning will help you match your inference strategy to your workload's actual characteristics. This kind of planning becomes critical as you move beyond simple chatbots toward sophisticated agentic architectures. An agent reasoning through a complex task might invoke an AI model multiple times per user request, with each call consuming tokens at varying rates.
In this post, we show you how to build performant and cost-effective inference systems on Amazon Bedrock , a fully managed service that offers a choice of industry-leading foundation models (FMs) along with a broad set of capabilities to build generative AI applications. We first look at what makes large language model (LLM) APIs different from traditional APIs. We then walk you through a practical three-step framework for capacity planning on Amazon Bedrock. You learn how to profile your workload, calculate your quota requirements, and select the right inference options to optimize both performance and cost. Finally, we explore how Bedrock's built-in features — like cross-region inference and prompt caching — automatically scale your effective capacity, and how custom patterns can extend these capabilities for specialized scenarios.
Let's start by examining how LLM API inference differs from traditional APIs — and why those differences matter for capacity planning.
What makes inference on LLM APIs different
API providers use rate limiting and quotas to manage system load, maintain service availability, and provide fair access across customers. For traditional APIs — e.g., payment processing, messaging, geolocation — quota management typically focuses on a single dimension: request count. You measure your peak request rate, add a buffer, and confirm your quota accommodates the total. Whatever variability exists in underlying resource consumption per request, the provider manages it internally.
Multi-dimensional quota management in LLM APIs
LLM APIs fundamentally change this model by introducing a critical second dimension: token consumption. Token-based limits exist because tokens are the fundamental unit of cost for LLM inference, and the variability in token usage is both extreme and significantly influenced by the customer. A document generation request may consume 50–100× more tokens than a classification task. Even within a single task type such as summarization, token consumption fluctuates based on input length, context augmentation, and output length.
This is why Amazon Bedrock—like most LLM API providers—structures quotas across both request count and token consumption. Amazon Bedrock tracks three metrics : requests per minute (RPM), tokens per minute (TPM), and tokens per day (TPD). RPM and TPM are measured within a one-minute window, while TPD enforces a daily ceiling. Token-based quotas combine input and output tokens—a request with a 1,000-token prompt and 500-token response consumes 1,500 tokens toward your TPM and TPD quotas. These quotas work as interdependent dimensions that shift based on workload composition — understanding their interaction is key to right-sizing your capacity. To view your quotas for a specific model on or request an increase, see Service Quotas documentation .
Token-based limits prevent high-consumption requests from monopolizing quota at the expense of other concurrent requests. Consider an application with 1,000 RPM and 200,000 TPM quota. A classification workload making 500 requests averaging 200 tokens each consumes 50% of both RPM and TPM—comfortable headroom. But shift that same application toward document generation at 50 requests averaging 4,000 tokens each, and the picture changes: RPM utilization drops to 5%, yet TPM hits 100% as shown in the figure below. The application reaches its TPM ceiling despite low request volume — illustrating why token consumption, not just request count, drives capacity requirements.
User requests vs. inference calls
It's important to distinguish between what your users trigger and what consumes your Amazon Bedrock quota:
- User request: What your end user initiates—sending a chat message, uploading a document, or triggering a report generation.
- Inference call: An individual API call to Amazon Bedrock that consumes quota.
The relationship is not always 1:1. A single user request can trigger multiple inference calls, creating a multiplier effect that varies by architecture:
- Agentic architectures: Variable multiplier
An AI agent reasoning through a customer service query might make 5, 10, or 20+ inference calls—analyzing intent, evaluating tool results, planning next steps, and generating responses as shown in the following figure. The count varies by query complexity and is determined at runtime.
- Pipeline architectures: Fixed multiplier
A document analysis pipeline might make exactly 3 inference calls per upload: entity extraction, risk assessment, and summary generation as shown in the following figure. Predictable, but still amplified.
Note: Amazon Bedrock quotas apply per inference call, not per user request. If your agent averages 6 inference calls per user message, 100 user requests per minute translate to 600 inference RPM. This 6× multiplier applies to both RPM and TPM, making it important to profile at the inference call level rather than the user request level.
The need for diverse inference options
Knowing how much quota you need is only part of the picture. The same LLM might power an interactive chatbot requiring sub-second latency, a batch document summarization job that can run overnight, and a business-critical agentic workflow that needs guaranteed capacity. Each has different requirements for latency, cost, and availability.
Amazon Bedrock provides multiple inference options to address these varying requirements. For real-time inference, you can choose from four service tiers:
| Tier | Best for | Characteristics |
|---|---|---|
| Reserved | Mission-critical, high-availability workloads | Guaranteed capacity with 99.5% uptime target. 1 or 3-month commitments. Overflows to Standard. |
| Priority | Customer-facing workflows requiring lowest latency | Fastest response times, premium pricing. No upfront commitment. |
| Standard | Everyday AI tasks, content generation, text analysis | Consistent performance. Default tier when no tier is specified. |
| Flex | Background tasks, agentic workflows, evaluations | Cost-effective pricing for workloads tolerant of longer processing times. |
To learn more about service tiers, see Service tiers for optimizing performance and cost . For workloads that don't require real-time responses, Batch inference offers the most cost-efficient option for processing large volumes asynchronously.
Getting your quota and inference tier selection right directly impacts both user experience and cost efficiency. The framework in the next section helps you avoid under-provisioning (which limits throughput) or over-provisioning (which increases spend unnecessarily).
A capacity planning framework for Amazon Bedrock
The following framework translates your application's requirements into concrete capacity decisions: how much quota you need and which inference options fit your workload. We work through a realistic example—an agentic customer service application—to illustrate each step.
Our example application handles three distinct workload types. Live chat is the most demanding: an AI agent assists customers in real-time, using tool calls to check order status, process returns, and answer policy questions. Each conversation triggers multiple inference calls as the agent reasons through it. Ticket summarization serves a different need. For complex tickets—whether escalated from the AI agent, accumulated through long email threads, or spanning multiple support interactions—the system makes a single inference call to generate concise summaries so human agents can quickly understand the context and history. Finally, weekly reporting runs batch analysis over conversation trends, resolution rates, and common customer issues for the operations team.
This mix—agentic real-time, single-call processing, and offline batch—reflects the workload diversity most production applications face.
The framework has three steps: profile your workload, calculate quota requirements, and optimize your inference strategy as shown in the figure below.
Note: Capacity planning is an iterative process. As your application evolves, usage patterns shift, or new features launch, revisit these steps to verify your capacity aligns with actual demand.
Step 1: Profile your workload
Since Amazon Bedrock enforces quotas at the inference call level, profile your workload based on inference calls rather than user requests—especially for workflows where a single user request triggers multiple inference calls.
Profile inference call types
Characterize each distinct type of inference call in your workload:
- Inference call type: What task does this call perform? (e.g., intent analysis, document summarization, tool results evaluation, response generation, etc.)
- Distribution: What percentage of your total inference calls does this type represent?
- Token consumption: Average input and output tokens per call.
- Latency requirement: How quickly must this call complete?
Note: For input token calculations, include system prompts, retrieved context, conversation history, tool definitions, etc. Output tokens may count at higher rates toward your quota for certain models. For more details, see How tokens are counted in Amazon Bedrock .
Enable model invocation logging on Amazon Bedrock to capture detailed metadata for each inference call in CloudWatch Logs, and instrument your application to track call types (e.g., via custom identifiers or tags). Query your logs to extract token consumption and latency metrics, then combine this with your call type tracking to calculate the distribution of each call type. For new applications without production data, test with realistic scenarios to generate baseline measurements. For existing applications, analyze actual production traffic patterns. For aggregate quota consumption trends over time, use Amazon CloudWatch metrics for Amazon Bedrock .
Identify your user traffic pattern
Understanding how your load varies over time helps you identify peak demand periods. Most workloads fall into one of these patterns:
- Consistent: Steady, predictable load with minimal variation throughout the day. Common for global 24/7 services.
- Cyclical: Regular daily or weekly patterns with predictable peaks and valleys. Examples include business-hours applications with lunch-hour rushes or month-end processing surges.
- Bursty: Baseline load with unpredictable spikes or sporadic usage with idle periods. Common in event-driven workflows or campaign-based processing.
- Emerging: New applications without established usage history, such as pilot projects or initial launches.
Example: Profiling our customer service application
Our application follows a cyclical pattern: steady user traffic during business hours (9 AM–6 PM) with a 3× spike during lunch hour (12–1 PM). Here's our inference call type breakdow
Our application follows a cyclical pattern: steady user traffic during business hours (9 AM–6 PM) with a 3× spike during lunch hour (12–1 PM). Here's our inference call type breakdow
o
| Call Type | Use Case | % Calls | Ave Tok (In) | Ave Tok (Out) | Lat Req |
|---|---|---|---|---|---|
| Intent analys. | Chat | 15% | 800 | 200 | <1s |
| Tool sel. | Chat | 20% | 1,200 | 100 | <1s |
| Tool eval. | Chat | 20% | 1,100 | 300 | <1s |
| Resp. gen. | Chat | 15% | 1,200 | 400 | <1s |
| Iter. | Chat | 15% | 900 | 600 | <1s |
| Summ. | Ticket | 15% | 2,200 | 800 | <5m |
Traffic breakdown: During steady hours, the application needs to handle 100 user requests per minute split between chat and ticket workflows. However, user requests don't translate one-to-one into inference calls—live chat conversations trigger multiple inference calls as the agent analyzes intent, selects tools, evaluates results, generates responses, etc., while ticket summarization requires a single call. Based on our profiled distribution, we need to accommodate approximately 350 inference calls per minute during steady state. During the lunch peak (3× multiplier), this scales to approximately 1,050 inference calls per minute.
Step 2: Calculate your quota requirements
Translate your workload profile into specific RPM and TPM requirements. Calculate both peak quota (to avoid throttling during highest demand) and baseline quota (to inform cost-effective inference decisions in Step 3).
Formulas
Aggregate across all your inference call types using these formulas:
Aggregate across all your inference call types using these formulas:
RPM = Σ (Inference calls per minute by type)
TPM = Σ (Inference calls per minute by type × Average total tokens per inference call)
Now, let’s calculate both peak and baseline quotas for our example.
Peak quota (lunch hour (12–1 PM)): As established in our traffic analysis in Step 1, lunch hour generates approximately 1,050 inference calls per minute. Weekly reporting is excluded—it's processed offline via batch inference.
While RPM is straightforward to calculate (1,050 total inference calls per minute), TPM requires a more detailed breakdown. Since different call types consume different token amounts—intent analysis uses 1,000 tokens while response generation uses 1,600—we can't simply apply an overall average. Instead, we calculate TPM contribution for each call type separately, then sum across all types.
The following table shows how the 1,050 peak inference calls distribute across call types based on our profiled workload, along with each type's TPM contribution.
| Call Type | Calls/Min at Peak | Ave Tok/Call | TPM Total |
|---|---|---|---|
| Intent analys. | 158 (15% of 1050) | 1,000 | 158,000 |
| Tool sel. | 210 (20% of 1050) | 1,300 | 273,000 |
| Tool eval. | 210 (20% of 1050) | 1,400 | 294,000 |
| Resp. gen. | 158 (15% of 1050) | 1,600 | 252,800 |
| Iter. | 158 (15% of 1050) | 1,500 | 237,000 |
| Summ. | 158 (15% of 1050) | 3,000 | 474,000 |
Calculated peak requirements represent your observed or expected maximum load. Adding buffer capacity accounts for natural variance and gives you headroom for growth. The appropriate buffer size depends on factors including your traffic predictability, tolerance for occasional throttling, and confidence in your estimates.
For our example, we apply a 25% buffer:
- Peak RPM with buffer: 1050 × 1.25 = 1,313 RPM
- Peak TPM with buffer: 1.69M × 1.25 = 2.1M TPM
Baseline quota (steady hours):
Since our lunch peak represents a 3× spike over steady-state traffic, our baseline is simply one-third of peak. Applying this ratio gives us approximately 438 RPM (1,313 ÷ 3, rounded up) and 700K TPM (2.1M ÷ 3, rounded up) for steady-hours operation.
Since our lunch peak represents a 3× spike over steady-state traffic, our baseline is simply one-third of peak. Applying this ratio gives us approximately 438 RPM (1,313 ÷ 3, rounded up) and 700K TPM (2.1M ÷ 3, rounded up) for steady-hours operation.
| Metric | Peak | Baseline | Gap |
|---|---|---|---|
| RPM | 1,313 | 438 | 875 |
| TPM | 2.1M | 700K | 1.4M |
The gap between baseline and peak quotas (875 RPM and 1.4M TPM) represents the additional capacity needed specifically during the 1-hour lunch peak.
Note: TPD quota in Amazon Bedrock defaults to TPM × 24 hours × 60 minutes. If daily token consumption approaches the TPD limit, requesting a higher TPM quota proportionally increase TPD. Additionally, for applications using multiple models, calculate requirements separately per model — quotas are model-specific.
Step 3: Optimize your inference strategy
With your quota requirements calculated, map inference calls to service tiers to optimize the latency-cost tradeoff.
Select service tiers for each inference call type
Amazon Bedrock provides four service tiers for real-time inference—Reserved, Priority, Standard, and Flex—plus Batch inference for offline processing. Your on-demand quota is shared across Priority, Standard (default), and Flex tiers, while Reserved tier and Batch capacity are provisioned separately.
When selecting tiers, consider the latency sensitivity of each inference call and how it contributes to overall user experience. A customer-facing response that users actively wait for might justify a higher tier to minimize latency, while an internal processing step or background task might perform adequately on a lower-cost tier. Testing your workload under realistic conditions is essential—measure the latency differences across tiers and determine which optimizations meaningfully impact your user experience.
For agentic workflows where multiple calls execute sequentially, remember that each call's latency contributes to the total response time the user experiences. A call that seems internal still affects when the final response arrives. Evaluate whether promoting specific high-latency calls to Priority tier improves the end-to-end experience enough to justify the cost.
Inference strategy for our example: For live chat, the response generation step streams output directly to the chat interface, making time-to-first-token critical for user experience. Applying Priority tier to this step can significantly improve when users see the response beginning to appear, warranting the cost premium. The intermediate agentic steps (intent analysis, tool selection, tool result evaluation, iteration) use Standard tier, which provides adequate performance without materially impacting the response time. Ticket summarization is an internal workflow where human agents can tolerate slightly longer processing, so Flex tier reduces cost without impacting productivity. Weekly reporting doesn't require real-time results, making Batch inference the clear choice for cost efficiency.
Reserved tier considerations
Reserved tier guarantees capacity at committed pricing, with automatic overflow to on-demand when demand exceeds the reservation. This works best for workloads that run continuously — you reserve at your baseline and let on-demand handle peaks. For our example, the application runs 9 hours per day with a 1-hour peak, so reserved capacity would sit idle 15 hours per day — pure on-demand is the better fit. A 24/7 global service with the same traffic shape, however, would benefit from reserving at baseline (700K TPM) and overflowing to on-demand during peaks.
Scaling beyond planned capacity
As your application grows, you'll encounter demand that exceeds your initial projections — a holiday shopping surge, a viral product launch, or a new feature driving higher engagement.
Amazon Bedrock's automatic scaling capabilities
Bedrock provides several features that automatically expand or optimize your available capacity without requiring code changes or manual intervention.
Cross-region inference automatically expands your available capacity across AWS Regions. You enable either geography-specific routing (US, EU, APAC) or global routing, and Bedrock selects the optimal Region for each request. During high-demand periods, Bedrock distributes load across regions with available capacity — without additional routing costs and with data remaining on the AWS network. For our customer service application, this means a viral product launch driving 3× volume is handled seamlessly — requests flow to a region with available capacity, maintaining the same user experience.
Prompt caching expands effective capacity and improves latency for workloads with repeated contexts. Bedrock caches prompt prefixes at designated checkpoints, with a time-to-live that resets on each cache hit. Tokens read from cache don't count toward your TPM limits and are retrieved significantly faster at reduced rates — delivering capacity, performance, and cost benefits simultaneously. In our ticket summarization scenario—where human agents review multiple tickets about the same product issue—caching the shared product context reduce both processing time and TPM consumption for subsequent tickets, allowing you to handle more tickets faster within your quota.
Custom patterns for advanced scenarios
These built-in features cover most scaling needs. For specialized requirements, custom patterns offer additional flexibility.
One approach is to architect your application to work with multiple models on Amazon Bedrock. Switching between models—such as from Anthropic Claude Sonnet to Haiku— lets you optimize for cost and latency across call types and provides an additional scaling path during peak demand. By designing your prompts and application logic to be model-agnostic or maintaining model-specific prompt templates, you can route requests to alternative models when needed. For detailed guidance on implementing multi-model architectures, see Multi-LLM routing strategies for generative AI applications on AWS .
For applications that use models across multiple providers, the Multi-Provider Generative AI Gateway on AWS provides centralized routing, retry logic, and usage policy management across providers including Amazon Bedrock.
Conclusion
Effective capacity planning for LLM workloads requires moving beyond simple "request more quota" thinking. The multi-dimensional nature of LLM inference—request-based and token-based quotas, variable token consumption, and multiplier effects from complex workflows—demands a structured approach.
The framework in this post helps you translate application requirements into concrete capacity decisions: profile your workload at the inference call level, calculate both peak and baseline quota requirements across RPM, TPM and TPD dimensions, and optimize your inference strategy by mapping calls to the right inference option.
Bedrock's built-in capabilities — cross-region inference and prompt caching — automatically scale your effective capacity beyond your initial projections. For specialized requirements, custom patterns like multi-model routing offer additional flexibility. Together with the capacity planning framework, these capabilities help your application perform efficiently across a wide range of demand scenarios.
Ready to get started? Review your current Amazon Bedrock quotas in the Service Quotas console , profile your workload using the capacity planning framework, and optimize your inference strategy to match your application's requirements. For applications already in production, use the Amazon Bedrock Usage Analysis CLI tool to analyze your historical token consumption patterns and identify optimization opportunities.
Learn more:
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article