The Boring Half of Every Agent's Prompt
I had been paying my model to relearn its own job, thousands of times a day. cachePoint was the parameter that finally let it remember.
I sat with the bill for a while before I started reading it properly.
The total was high, and I had expected it to be high — a heavy month with three planners on the copilots, more voice calls than I'd budgeted for, a couple of demos that ran longer than they should have. The total wasn't the surprise.
The share was.
Most of the spend wasn't the work. It was the setup. Every time one of our agents picked up a turn — and there had been, give or take, forty thousand of those that month — we had handed the model a five-thousand-token system prompt and a tool catalog it had already seen the call before. The bill had charged us for the privilege of telling it again.
I tried, briefly, to picture what forty thousand re-readings of the same four-thousand-word document would look like if a human were doing it. I gave up.
What I went looking for the next morning wasn't a way to cut costs. It was a way to understand why a cost like that was even possible. The answer turned out to be a feature in the Bedrock docs I had read past three times without seeing properly — cachePoint. Once I understood what it was, the bill came down. But that wasn't the part that changed how I work. The part that changed how I work was realizing the system prompt and the user's question were not the same kind of thing, and shouldn't be treated that way.
This is what cachePoint actually does, and what I do with it now.
What cachePoint is
CachePoint is a marker you drop into the Converse API at the end of any section of input that doesn't change between calls — a system prompt, a tool catalog, a long reference document. Bedrock holds the model's processing of everything up to that marker in a cache, keyed on the exact bytes of the prefix. The next call that arrives with the same prefix doesn't re-process it. It reuses the cache.
The model still runs end-to-end. It still produces an answer to the actual question. The part it skips is the part it has already done — the reading, the parsing, the building of context.
Here is the shape, on a typical agent call:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
import boto3, json
bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")
SYSTEM_PROMPT = open("checkout_system_prompt.txt").read() # ~4,200 tokens
TOOL_SPECS = json.load(open("checkout_tools.json")) # ~1,100 tokens
response = bedrock.converse(
modelId="us.anthropic.claude-sonnet-4-6-v1:0",
system=[
{"text": SYSTEM_PROMPT},
{"cachePoint": {"type": "default", "ttl": "1h"}},
],
toolConfig={
"tools": TOOL_SPECS + [{"cachePoint": {"type": "default", "ttl": "1h"}}],
},
messages=[
{"role": "user", "content": [{"text": "Hold two of SKU-714 for me."}]},
],
)
# The cache lives in the usage payload, not in your imagination.
print(response["usage"])
# {
# "inputTokens": 18, # the user's actual question, this call
# "cacheReadInputTokens": 5298, # the prefix served from cache
# "cacheWriteInputTokens": 0, # ...because the cache was already warm
# "outputTokens": 92,
# }Two markers. Both with a one-hour time-to-live. Everything above the first marker — the system prompt — gets cached. Everything above the second marker — the tools — gets cached too. The user's message lives below both markers, where it belongs, because that is the part of the call that changes every turn.
The shape of the response is the part worth staring at.
cacheReadInputTokens is what didn't pay full price. cacheWriteInputTokens is the small premium paid the first time, when the cache was being built. Once you start logging both, the feature stops feeling like magic and starts being measurable.The three things worth caching
Almost everything I cache in production falls into three buckets.
The system prompt. The obvious one. Most non-trivial agents carry a long system prompt explaining the role, the tools, the tone, the constraints, the do-nots. None of it changes call to call. It lives above the first cachePoint and stays warm as long as the agent is being used.
The tool catalog. This is the one most builders miss. Tool definitions in the Converse API are tokens like any other — a catalog with nine tools and decent descriptions easily sits at 1,000 to 1,500 tokens on its own. Putting a cachePoint at the end of the
tools array caches that part too, separately from the system prompt. You get the benefit even on agents whose system prompts are too short to qualify on their own.Reference documents. The case the docs lead with, and the one I use least often — but when it fits, it's the dramatic one. A retrieval-augmented agent that answers many questions against the same long document can cache the document once and pay almost nothing per question after the first. If you've built RAG before, you've felt the wasted re-reading. cachePoint is the answer to it.
What these three have in common is the only thing that matters: they don't change. Whatever sits above a cachePoint must be byte-identical from one call to the next. Timestamps, user IDs, session IDs, the user's actual question — all of that has to live below.
The latency side of the story
The cost side gets all the airtime, because cost is easy to write about. The side that gets less attention, and matters at least as much in some workloads, is latency.
For my voice agents in particular — the ones using Nova Sonic for real-time calls — the time the model spends processing a five-thousand-token system prompt is the time the customer spends listening to silence. Cutting that processing takes the cold-call response from "noticeably robotic" to "almost natural." AWS publishes a headline number — up to 85% latency reduction on cached portions — and on the workloads I've measured, that's roughly what I see.
If you build anything where first-token-time matters — voice, streamed chat, live retrieval — cachePoint is worth turning on before you start optimizing anything else.
TTL: five minutes, or one hour
The default cache lifetime is five minutes. Plenty for a chatty workload where turns arrive every few seconds. Useless for a sub-agent that gets invoked once every twenty minutes across a planner's shift.
In January 2026, AWS added a one-hour TTL option for the Claude 4.5 family, and support has since broadened across the newer Anthropic models on Bedrock. You opt in by adding
"ttl": "1h" inside the cachePoint object. Most of my agents run on the one-hour TTL now, because most of my agents are bursty rather than continuous.The rule I use: if you expect more than one call inside any five-minute window, the default is fine. If your calls are sparser than that but still come back within an hour, opt in. Below that frequency, the write premium isn't worth it.
Designing prompts for the cache
Once I understood what cachePoint was doing, the way I write prompts changed.
The discipline is simple but takes a little un-learning. Anything stable goes above the marker. Anything volatile goes below. A current timestamp, a per-user greeting, a session ID, the user's latest message — these are all below-the-line content, even if they feel like they belong at the top of the prompt for readability reasons.
The one that bit me first was a system prompt that opened with the date. Today is Tuesday, March 4. A friendly, helpful thing to tell the model. Also a guarantee that the cache would never hit, because the prefix changed every twenty-four hours. The fix was to move the date down into the user-message section, where it could change without dragging the cache with it.
This kind of thinking — what is stable, what is volatile, where does the boundary go — is now part of how I draft every prompt. The cache doesn't only save tokens. It makes the structure of the prompt more honest about what is reused and what is fresh.
What I'd tell past-me
The cache is keyed byte-for-byte on the prefix. A single whitespace added at the top of the system prompt during a cleanup pass is enough to take a cache hit rate from 95% to roughly zero. The model still answers correctly, so nothing alarms. The bill just quietly drifts back to what it was before. The defense is logging — version the system prompt, log a hash of the cached prefix on every call, and alarm when cache hit rate falls below a threshold. Without that, you can lose the feature and not notice for weeks.
There is also a minimum prefix length. Across the current Claude models on Bedrock — including Sonnet 4.6 — a cache checkpoint requires at least 1,024 tokens above it. If your prefix is shorter than that, the call still succeeds, but nothing actually gets cached and no error tells you. You can place up to four cache checkpoints in a single request, and the total cached portion can run up to 32K tokens. For an agent whose system prompt sits at, say, 800 tokens, you have two choices — accept that this agent won't benefit, or pad the system prompt with stable content like schema definitions, glossary entries, or reference text until it clears the minimum. I have done both. The padded version stops feeling strange after about a week.
One more ordering rule worth knowing about: if you place cache checkpoints with different TTLs in the same request — say a one-hour TTL on the system prompt and a five-minute TTL on a per-conversation context block — the longer-TTL checkpoint must appear earlier in the request than the shorter-TTL one. Bedrock rejects the call otherwise.
And the savings live in the usage payload, not in your gut. Ship
cacheReadInputTokens and cacheWriteInputTokens into CloudWatch as custom metrics on every call. Watch the ratio for a week. That number, much more than any monthly invoice, tells you whether the feature is doing the work you think.And yes — the bill
Back to the Tuesday morning I started with.
The hard question had, in fact, been asked. Quietly, in a thread, three weeks before that bill landed. Are we sure we need all of these agents? I had answered with a slide deck, which is what you do when you don't yet have a better answer. The bill that arrived that Tuesday made me wish I had spent that hour in the Bedrock docs instead.
Four weeks after I added cachePoints across the system, the next bill came in at roughly half what the previous one had been. Cache reads ran about a tenth the cost of standard input tokens for our models. Cache writes cost about 25% more, paid only on the first call into a cold cache. On our call volume, the math turned over in two or three calls. After that, the savings were no longer interesting in isolation — they were structural.
The thing that changed downstream was quieter, and to me more interesting than the bill itself. The agents we had been quietly thinking about retiring — the ones paying their way but only just — stopped being on the bubble. We added a ninth specialized agent two weeks later, the one I had been holding off on because the cost story didn't quite work. It works now. The bill is still smaller than before I started.
The hard question stopped being asked.
What cachePoint is really teaching
CachePoint isn't a cost trick. It's a piece of model infrastructure that quietly enforces a discipline most of us had been avoiding — separating the stable part of a prompt from the volatile part, and naming where the boundary is. The cache rewards that discipline. The bill happens to drop as a side effect. The latency drops with it. And the prompts get easier to reason about, because the question what is reused and what is fresh has to be answered every time.
If your agents have grown long system prompts, heavy tool catalogs, or repeated reference passages, add a cachePoint and watch the usage payload for a few days. The savings will show up. So will some prompt habits you'd been getting away with.
If you have been using cachePoint already and have run into something I haven't, I'd like to hear it.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article