AWS Builder Center

AI economics beyond caching, a practical guide to token optimization

From token caching to advanced context engineering, this article lays out, how to reduce your input token consumption for generative ai workload without sacrificing model performance.

With growing adoption of agentic workloads, some organizations with limited margins start asking about token economics. While the adoption of generative and agentic solutions keeps rising, many customers want to improve cost performance while still scaling.
But cost only matters relative to outcomes. Spending a million dollars to save ten is not a cost problem. The goal of cost optimization is to save cost without slowing adoption. To achieve this, this post covers caching, context engineering that improves cache hits, and how to measure the results.

How prompt caching works

Many system designers think of caches as binary: hit or miss, then fall through to the underlying source. This works for static content and API’s producing deterministic results. But prompts are more dynamic. You rarely send the exact same request twice. A prompt stacks system instructions, environment variables, generated content like file lists, and sometimes serialized data. The mechanic of caching here needs to be more dynamic than full-request hit-or-miss.
In order to understand this partial caching behaviour you need to understand checkpoints and the Key-Value-cache (KV-cache) of a model. This KV-cache caches the prefix of incoming prompts to retrieve the models internal activation state from a caching component, instead of its calculation.
Implementation for caching is straightforward because most clients and SDKs for Amazon Bedrock already handle checkpoint placement and Amazon Bedrock handles the caching itself. Some clients use naive rules (like checkpoints every N input tokens), others apply heuristics to try and find the most efficient boundaries. If you use a framework, check its docs for which strategy it uses and whether it exposes a configuration. If you write against the Bedrock API directly, you have to place  cachePoint  markers yourself. To verify that a checkpoint was effectively set, inspect the  TokenUsage  in the response:  cacheWriteInputTokens  on the first call and  cacheReadInputTokens  on a subsequent call to see if the prefix was cached.
Bedrock prompt caching hashes the prefix. This prefix should include all tool definitions, system prompt, personas and static instructions - but typically not the user prompt. It then retrieves the cached KV state on subsequent identical prefixes. The pricing structure shows why even a single cache hit pays off the writing cost.
Token type
Cost relative to input
Uncached input
1x
Cache write
1.25x
Cache read
0.1x
Reading from the cache once and means you pay 1.35x instead of 2x. This effect only compounds in multi-turn conversations.
Many models default to a 5-minute cache time to live(TTL). Longer TTLs are available at different rates. Check the  Amazon Bedrock pricing page  for current numbers. The optimal TTL to choose varies by workload and warrants some experimentation, depending on user and agent behaviour.
To keep hit rates high, engineer a stable context, especially at the top of the prompt. Four habits:
  1. Stable content first. Tool definitions, then system prompt, then static context (personas and instructions), then tool output, and only then the varying user message. If a timestamp or request ID sits near the top, no subsequent cache request will hit.
  1. Place  cachePoint  on the last block that stays identical, not on the incoming user message. The key for the kv-cache is the content up to the breakpoint and caches everything before it. If your framework sets checkpoints for you, verify they land where you expect by inspecting the request payload and the  TokenUsage  response.
  1. Mind the TTL. Default is 5 minutes. Some Claude models on Bedrock support a 1-hour TTL via  "ttl": "1h"  on your  cachePoint . Check each model's individual model card for supported TTLs. If your sessions have idle gaps longer than the TTL (common for human-in-the-loop agents), the TTL decides whether you pay cache reads or cache writes on every turn.
  1. Use multiple checkpoints. Claude models on Bedrock allow up to 4 cache checkpoints per request. Place them at stable boundaries. Each checkpoint has a model-specific minimum token count; see the Bedrock  prompt caching supported models docs  for current thresholds. A checkpoint below the minimum fails silently: your request succeeds, the prefix is not cached, and you pay the full input rate.
Two constraints are worth knowing:
Cross-region inference. When using cross-region inference endpoints your prompts might get routed to another region, if a regions faces high inference traffic. This is great for inference availability but the KV-cache is region-specific. Meaning, that cross-region inference can have a negative impact on cache hits exactly at peak scaling. Region specific endpoints might face scaling issues in this case - so it is a case-to-case decision.
Batch inference does not cache. Prompt caching only applies to on-demand endpoints. If you run bulk generation via the batch API, caching is not available.
For more depth on the API and checkpoint patterns, the AWS Machine Learning blog on  prompt caching  covers the code. This post will focus on other practices to improve cache hits and cost optimization.

Stable context: engineering tool output

To engineer a stable context, keep the structure above (tools, system prompt, tool output, user message) and make sure each part is consistent. Tool definitions and system prompts rarely change within a 5-minute or 1-hour TTL. Tool output and user messages are where most cache misses come from.
For tools used across agents or across your organization, optimizing output for cost pays back. Even small changes in tool output can break caching for every downstream user, human or agent.
Every  ls ,  find ,  git diff ,  grep ,  curl , or  cat  returns raw output into your context. If your agent runtime keeps full tool output in model-visible history, as many do by default - if you don’t explicitly configure history compression, that output rides along in every subsequent turn. By turn twenty you carry the output of every tool call from turns one through nineteen.
Tighter tool invocations reduce this load by returning results instead of raw data:
  •  ls  →  ls -1 | head 
  •  git diff  →  git diff --stat 
  •  grep -r  →  grep -l 
Tools like  rtk  strip CLI chatter that fills context windows without adding value, sometimes invisibly to a human reader.
The same principle applies well beyond the shell. An agent that plans or repairs a trip talks to availability, fare rules, loyalty, payment, and notification services. Those responses are JSON built for machines, not readers. A supplier response with forty fields when the agent needs three carries the other thirty-seven into every subsequent turn. Trimming responses at the integration layer, before they reach the model, is often the single largest reduction available.

Context rot and context engineering

Long-running agents hit context decay well before the window fills. Modern frontier models handle context up to a million tokens, but the larger the context, the less clear it becomes which parts matter for the current task.
Neither a model nor a human can judge relevance in a filled context if the information inside it is inconsistent. More context is not always better, especially when parts are stale or contradictory.
 Lost in the Middle  found that models attend most to the beginning and end of their input and degrade on information placed in the middle. Performance decreases as input grows longer, even for models built for long contexts. A larger window postpones the problem rather than solving it.
For cost, this matters twice. Rot produces worse answers and it introduces noise. It bloats context, which drives cost per turn up. Bad answers lead to retries, retries reload the same bloated context, and the problem compounds.
Three fixes that work:
  1. Pin the task description. The top-level objective should at the front of the dynamic context and never be evicted. If your framework compacts context automatically, verify the task itself is not being summarized away.
  1. Maintain an explicit dead-end list. When summarizing prior turns, capture what did not work and label it clearly: "Do NOT retry these approaches: X, Y, Z." Failed approaches that are not flagged as failed get retried. This is the single most common loop in agent trajectories.
  1. Checkpoint-resume before you hit the wall. When context nears capacity, do not wait for a compact mid-tool-call disaster. Trigger  /compact  (Claude Code or Kiro-Cli) or compact the context history manually when using the API.
Consider this practical example: A disruption management agent needs data from many source systems to build an alternative travel plan across a batch of passengers. Managing that context deliberately means three things: (a) each source system returns only the information the agent needs, (b) contradictions get resolved before the model sees them (or in a subagent that does not share history and internal state), and (c) prior-turn summaries drop passenger records that have already been re-accommodated. Without those three, the agent burns tokens rereading data it has already acted on.

What gets measured

Measuring the productivity of knowledge workers has been the defining management task of the last century. Teams often fall back to what is easy to measure (lines of code) or what is only weakly correlated with output (story points). These are proxies.
Proxies get more dangerous with LLMs. Models generate output at scale; validating that output usually falls back to humans. This right-shifts your bottleneck instead of removing it. Without a way to attribute value to output, you cannot tell whether a change to prompts, models, or tools improved results. You cannot set task budgets. You cannot manage token cost.
To make token optimization count, you need to understand what the task is worth. This is not a new problem. It is the basics of management: understand the nature of the work, its requirements, and its outcomes. For each task you hand to an agent, ask what defines a good outcome, why the work is done at all, and only then what it can cost.
This is not a technical challenge anymore: Every Bedrock invocation returns a  TokenUsage  object:  inputTokens  for uncached input,  cacheReadInputTokens  for cache hits,  outputTokens  for newly generated tokens,  cacheWriteInputTokens  for tokens stored in the cache, and  cacheDetails  for finer breakdown. Enable model invocation logging and those numbers land in CloudWatch Logs next to the caller identity, so spend becomes attributable per application or per user.
Tag each invocation with a task type and a terminal outcome, then divide cost by successful terminal outcomes instead of by sessions or tokens.
What counts as a successful outcome depends on the workload: a booking that completed and did not get refunded within 24 hours, a support case closed without human intervention that the customer did not reopen within a week, a disruption rebooking accepted by the passenger. Define it explicitly per workload. If you cannot define it, the workload is not ready for cost optimization yet.
Tokens per session will always be a moving target. It rises when users engage longer or when you add a tool. It can move in either direction when the agent gets more capable: more tool calls per turn, or fewer turns to reach the outcome. Cost per completed outcome does not drift with these variables. It gives you a stable number to optimize against.

Takeaway

Cost per outcome, not cost per session, is the only number worth optimizing. Everything in this post feeds into it:
  • Prompt caching cuts input cost per turn.
  • Stable context and trimmed tool output keep the cache hot across turns.
  • Context engineering reduces retries, which is where most compound cost lives.
  • Per-invocation tagging in CloudWatch turns spend into a per-workload number you can act on.
Two questions to carry into architecture reviews and pull requests:
  1. Define, in quality and numbers, what a successful execution of an agent or llm task looks like.
  1. Measure future changes against these numbers. If a prompt change, additional too or model change does not improve these measurements. Consider it a failed experiment.
If you cannot answer (1), no amount of caching will save you. If (2) points to more tokens without more outcomes, the change is not worth shipping.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article