The Bedrock Bill Nobody Warns You About: Choosing On-Demand, Batch, or Provisioned Throughput
Most Amazon Bedrock projects start the same way: you grab a model ID, call Converse , ship a demo, and everyone's happy. Then the invoice arrives, and the number is two or three times what the pricing page led you to expect. The model rate didn't change — you just picked the wrong inference mode for your workload and paid retail for something you could have bought at a discount.
Bedrock gives you three ways to buy inference, and the gap between the cheapest and most expensive path for the same tokens can be 50% or more. Here's how I decide, and a framework you can steal.
The three modes, honestly
On-Demand is the default. You pay per input/output token (roughly $0.035 per million input tokens for Amazon Nova Micro up to ~$10 per million for the top Claude Opus tier), with no commitment and no minimum. It's elastic and simple — but it's also subject to account-level quotas and throttling, so it's the least predictable under real traffic. Great for spiky, interactive, or early-stage workloads where you can't forecast volume.
Batch Inference is the mode teams forget exists. You drop a JSONL file of prompts into S3, kick off an async job, and results land back in S3 — typically within 24 hours — at 50% off On-Demand rates for supported Anthropic, Meta, Mistral, and Amazon models. If your work isn't interactive — nightly document classification, bulk summarization, embedding backfills, eval runs, synthetic data generation — batch is the single highest-leverage cost decision you can make. Half price, zero code rewrite beyond the job wrapper.
Provisioned Throughput reserves dedicated capacity measured in Model Units (MUs), each delivering a guaranteed tokens-per-minute ceiling. You pay hourly whether or not you use it — published rates run roughly $21–$50 per MU-hour depending on model — in exchange for guaranteed throughput and no rate limiting. You can buy it with no commitment, or lock a 1-month or 6-month term for a steeper discount. This is for sustained, high-volume, latency-sensitive production traffic where On-Demand throttling would break your SLA.
A decision framework
Ask three questions, in order:
- Is the workload interactive? If no — if a human isn't waiting on the response — default to Batch and take the 50% discount. This alone catches a huge share of real spend.
- Is traffic sustained and predictable? If you're running high, steady volume around the clock and On-Demand throttling threatens your latency, model out the math: once your hourly token consumption reliably exceeds what an MU delivers, Provisioned Throughput (ideally a committed term) usually wins on both cost and reliability.
- Everything else? Stay On-Demand. Spiky, low-volume, or still-finding-product-market-fit traffic doesn't justify a commitment, and you keep full elasticity.
The trap is treating this as a one-time architecture decision. It's a portfolio: a mature system often runs interactive traffic On-Demand, offloads nightly jobs to Batch, and reserves Provisioned Throughput only for the one high-QPS path that actually needs it.
Two levers that stack on top of everything
Before you commit a dollar to capacity, pull these — they apply across modes:
Prompt caching (GA since 2025) delivers up to 90% cost reduction on cached tokens and up to 85% latency improvement for repeated context. If you're resending the same system prompt, tool definitions, few-shot examples, or a document across turns, cache anything over ~500 tokens. For RAG and agent loops — which resend the same instructions constantly — this is close to free money.
Cross-region inference profiles spread load across Regions so you stop hitting single-Region capacity walls. Global inference profiles go further, unlocking a larger resource pool at roughly 10% lower cost than geographic cross-Region inference, and they still support prompt caching, Batch, Guardrails, and Knowledge Bases. If you're fighting throttling on On-Demand, a global profile is often the fix before you jump to Provisioned Throughput.
What I'd actually do
On a new project, I start 100% On-Demand with prompt caching on from day one and a cross-region profile configured — that's the cheapest predictable baseline with no commitment. Then I instrument token consumption per code path. Within a couple of weeks the data tells me which paths are batchable (move them, bank the 50%) and whether any single path is sustained enough to justify an MU. I never buy a 6-month commitment on a guess; I buy it when the usage graph makes the decision for me.
The mistake I see most often isn't picking the "wrong" model — it's paying On-Demand retail for workloads that were never interactive in the first place.
How are you splitting your Bedrock traffic across modes today — and has anyone actually modeled the break-even point on Provisioned Throughput, or is it still a gut call?
#AWS #Bedrock #GenAI #MachineLearning #CloudCost #FinOps #AIEngineering
Loading article