Your AI bill is rising while token prices fall. Six levers to fix it on Amazon Bedrock.
Per-token prices keep dropping, yet AI bills keep climbing. Here is how to see where your Amazon Bedrock spend goes, and six levers to cut it, in order of effort: prompt caching, batch inference, model routing, RAG tuning, semantic caching, and model customization. Last updated: November 2026. Figures reflect AWS documentation and pricing as of this date. Features, model support, and prices change often, so check the Amazon Bedrock pricing page and model cards before you plan around a number.
Your AI bill is going up, and per-token prices are going down. Both are true at the same time.
Cheaper tokens make it easier to say yes to new use cases. Agents call models many times per task. Each new workload adds volume. The bill grows because the program is working, and a single "Amazon Bedrock" line on the invoice tells you nothing about which team, application, or agent is driving it.
This post covers how to fix that on AWS: first make spend visible, then apply six levers, starting with the ones that need only configuration.
With thanks to Laura Tacho and Rajdeep Banerjee at AWS for their contributions to this work.
By the numbers
Lever | What AWS reports |
|---|---|
Prompt caching | Up to 90% lower cost on cached input tokens, and up to 85% lower latency, for supported models |
Batch inference | 50% lower price than on-demand, for select models |
Intelligent Prompt Routing | Up to 30% lower cost without compromising accuracy |
Model distillation | Up to 75% less expensive and up to 500% faster, with less than 2% accuracy loss for use cases such as RAG |
Semantic caching (Amazon ElastiCache benchmark) | 72.5% cost savings at a 0.90 similarity threshold |
These figures come from different sources and conditions. They overlap and do not add together.
Step 0: Find out where the money goes
Every lever depends on knowing which team, application, and workload is spending tokens. Without that, you cannot tell whether a change worked. Put attribution in place before you change anything, so each change has a baseline.
Amazon Bedrock supports attribution at three levels:
1. Application inference profiles. Create one profile per workload and attach cost allocation tags such as team, cost center, and environment. Invoke models through the profile so usage and cost are recorded against those tags.
2. IAM principal attribution. Bedrock can attribute billed inference cost to the IAM identity that made the request, in AWS Cost Explorer and the Cost and Usage Report.
3. Per-request detail. Billing data is aggregated, so per-prompt detail comes from model invocation logs, where each call is its own record with its own token counts.
A gotcha for shared gateways: if every request goes through one service role, every call looks like the same caller. Have the gateway assume its role per user or per workload, or pass session tags, so attribution survives the hop. Also activate your cost allocation tags in the Billing console; tags apply from activation onward, not to past usage.
Measure outcomes, not just tokens. Cost per invocation tells you whether individual calls are efficient. Cost per successful business outcome, counting every call, retry, and tool call behind it, tells you whether the program is economical.
Lever 1: Prompt caching
Many requests resend the same long content: system instructions, tool definitions, reference documents. Prompt caching lets Bedrock reuse that repeated prefix instead of processing it again.
· Two modes. With explicit caching, you mark cache checkpoints in the prompt. With implicit caching, Bedrock reuses eligible prefixes automatically on select models such as Amazon Nova. Anthropic Claude models use explicit caching, so requests without cache controls are not cached. Check the model card for your model.
· Put static content first and dynamic content last. Order the prompt as tools, then system, then messages. If anything that changes per request sits above the checkpoint, the cache misses.
· Watch the hit rate. Bedrock publishes CacheReadInputTokenCount and CacheWriteInputTokenCount to Amazon CloudWatch. A low hit rate on a stable workload usually means the checkpoint sits after content that varies per request.
· Check the pricing. Cache reads are billed at a lower rate. On some models, cache writes cost more than standard input. Many models use a 5-minute time to live, and some support longer.
Lever 2: Batch inference
Nightly reports, document processing, and evaluation runs can wait. Bedrock batch inference runs them asynchronously at 50% of the on-demand price for select models. You submit a JSONL file in Amazon S3, and results land in S3.
Completion is not guaranteed within a fixed time, so keep anything user-facing on the real-time path. Note that prompt caching is not supported with the batch inference API, so the two levers apply to different requests.
A quick audit: list your workloads by how fast they actually need a response. Anything that can wait an hour or more is a batch candidate.
Lever 3: Route each task to the right-sized model
Only some tasks need a frontier model. Many requests are structured, formulaic, or retrieval-based, and a lighter model handles them well. Validate quality on your own workload before you shift traffic.
You can act on this in two ways:
4. Model tiering. Decide per task type which model handles it, and call that model directly.
5. Amazon Bedrock Intelligent Prompt Routing. The router predicts response quality for two models in the same family and sends each request to the better balance of quality and cost. You use the router ARN where you would use a model ID. It is optimized for English prompts, and supported models vary, so check the user guide. Check the pricing page for any routing charge on top of model usage.
Review your routing regularly. Smaller models keep improving, so a split that was right six months ago probably sends too much traffic to the expensive model today.
Lever 4: Send less retrieved context
RAG is a common source of input token growth. Amazon Bedrock Knowledge Bases default chunking uses 300-token chunks with 20% overlap, so every extra chunk adds roughly 300 tokens to every query.
· Metadata filtering narrows the search before similarity search runs.
· Hierarchical chunking matches on small chunks but returns their parents, so you retrieve fewer overlapping fragments.
· Reranking retrieves a wider candidate set and keeps the best few. A common starting point is to retrieve 20 to 50 and keep 3 to 5, then tune on your own data.
As an illustration, retrieving 10 chunks of about 300 tokens sends about 3,000 context tokens. Reranking down to the best 3 sends about 900, a 70% cut for that query. Reranking has its own per-query charge, so with a low-cost model, rerank for answer quality rather than cost.
Check what each agent receives. In multi-agent workflows, passing the full history or case file to every agent multiplies input cost by the number of agents. Give each agent only the context its step needs.
Lever 5: Semantic caching
Prompt caching reuses identical prefixes. Semantic caching goes further: when a new question means the same thing as one already answered, the application returns the stored answer and skips the model call.
Amazon ElastiCache for Valkey (version 8.2 and later) supports vector search at no additional cost beyond node pricing. In an AWS benchmark on 63,796 real chatbot queries, a 0.90 similarity threshold gave a 74.5% cache hit rate, 92.3% accuracy, and 72.5% cost savings, and average latency fell by about 72%.
· Start strict. AWS recommends a threshold of 0.90 to 0.95, lowered gradually while you monitor accuracy.
· Use a shared, persistent store. A cache that lives in each application instance misses under concurrent load, so pilot savings will not carry into production.
· Plan for stale answers. Set time to live to match how often the source content changes, and build a way to clear affected entries.
Lever 6: Model customization and distillation
Customization moves high-volume, recurring tasks to a smaller model trained for them. Bedrock supports supervised fine-tuning, reinforcement fine-tuning, and distillation. Distillation trains a smaller student model from a larger teacher model's outputs; supported pairs change over time, and distillation is not currently available for Anthropic models on Bedrock.
This pays off only on stable, high-volume workloads. Work out break-even first: a $5,000 training run that saves $400 a month breaks even around month 13. Count teacher response generation, training, and custom model storage. Refresh on a schedule, and gate each new version on an A/B comparison.
The order to pull them
Effort | Levers | What it takes |
|---|---|---|
Configuration only | Prompt caching, batch inference | No new infrastructure |
Architectural change | Model routing, RAG tuning | Model ID or router ARN change, retrieval changes |
Infrastructure or training | Semantic caching, model customization | New infrastructure or a training run, justified by stable, high-volume workloads |
Guardrails: cut cost without cutting quality
6. Baseline first. Record cost and quality before you change anything.
7. Test on real traffic, using held-out production queries rather than synthetic benchmarks.
8. One lever at a time per workload, so you can attribute the result and roll back.
9. Watch latency too. Track time to first token, so a cheaper model or tighter retrieval does not slow a user-facing flow.
10. Treat caches and logs as sensitive data. They hold real prompts. Encrypt at rest, set time to live, and isolate tenants.
Checklist: start this week
· Create a tagged application inference profile for your highest-spend workload
· Add a cache checkpoint after your static system prompt and watch the hit rate
· Move one non-urgent job to batch inference
· Check what each agent in your longest chain actually receives
· Record cost per business outcome before and after
Measure the result, then decide whether you need the next tier.
Resources
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article