The Cheapest Price Per Token Can Be the Most Expensive Way to Run an LLM

I keep watching the same scene play out in architecture reviews. A team announces they cut their inference bill by moving an open model off an API and onto their own GPU instances. The price per token on the slide is a fraction of what they paid before. A few weeks later the real bill lands higher than where they started, and the room goes quiet because nobody can explain it.

I can explain it. They compared the one number that rarely decides the cost and ignored the ones that do.

When someone says “we run our own LLM,” that sentence hides a decision most people never make on purpose... They pick the model they want, then inherit an operating model they never evaluated. This week is about that second choice, the one underneath the agent, underneath the prompt, where the model actually runs and answers requests. The industry has a name for that layer. It is called serving, and it is where your GenAI bill actually lives.

“We run Llama” sounds like one decision. It is three, and they do not share a cost model. The gap between them is the gap between a bill you can forecast and a budget meeting nobody wants to sit in. The fastest way to tell them apart is to ask one question. What are you actually paying for?

The first is managed inference. You send a request to a hosted endpoint, you pay per token, and you never touch a GPU. Amazon Bedrock on-demand is the clearest example. Picture an internal support assistant that is busy during business hours and quiet overnight. You pay for the tokens it generates while people are using it, and at 2am, when nobody is asking it anything, the meter reads zero. The provider absorbs the cost of the idle hardware. You are renting tokens, and only the ones you use.

The second is dedicated capacity. Here you stop renting tokens and start renting a reservation. You commit to a block of guaranteed throughput, and the provider holds that capacity for you whether you fill it or not. Amazon Bedrock Provisioned Throughput and Amazon SageMaker real-time endpoints both sit here. The model still runs on the provider’s side, so you are not managing servers. What changed is the meter. A Provisioned Throughput Model Unit costs you the same in an hour where you push a million tokens through it and an hour where you push nothing. That is the right trade when your volume is steady and high, or when you need throughput the shared on-demand pool will not guarantee. It is the wrong trade the moment your traffic goes quiet, because the reservation keeps billing through the silence.

The third is self-hosted serving. Now you are not renting tokens or a reservation. You are renting raw hardware, and everything running on it is yours to operate. You take open weights, a fine-tuned Llama for instance, and you run them on your own fleet with a serving engine like vLLM doing the work underneath. This is vLLM on Amazon Elastic Kubernetes Service (EKS) or Amazon Elastic Compute Cloud (EC2). You pay for the GPU instances by the hour, every hour they are up, no matter how many tokens pass through them. On paper this gives the lowest price per token of the three. In practice that price is a division problem: instance cost divided by tokens served. Keep the hardware full and the number is tiny. Let it sit half empty and you are paying premium rates to run a model you could have rented for less. This path fits the team that needs a specific open model, full control over where the data sits, and enough sustained volume to keep expensive cards busy.

So the same five words, “we run our own model,” hide three different lines on your invoice. In the first you rent tokens. In the second you rent a reservation. In the third you rent the silicon and operate everything yourself. Three meters, three failure modes, and a different person who gets the call when each one breaks at 3am.

If you last looked at Bedrock a year ago, you remember one on-demand price per model. That is no longer the shape of it.

The managed path is now a dial. You set it per request, with a single parameter, and it trades speed against cost. Same model, same call, different bill depending on where you set the dial.

Standard is the default, the baseline rate you get if you set nothing.

Priority is the fast lane. It costs roughly 75 percent more and runs up to 25 percent faster, and your requests jump ahead of others when the system is busy. This is where you put a customer-facing checkout assistant, because a slow answer there can cost you the sale.

Flex is the slow lane. It runs at about half the Standard rate, in exchange for latency measured in minutes instead of seconds. This is where you put the overnight job that summarizes yesterday’s tickets while everyone is asleep and nobody is watching the clock. If the work can wait even longer, Batch gives you that same discount by taking a whole file of requests and processing it offline.

Reserved sits off to the side of the dial. Instead of paying per request, you pre-buy a guaranteed block of throughput for one or three months, so a traffic spike cannot throttle you. Go over what you reserved and the overflow falls back to Standard.

The shift is small to describe and large in practice. The managed path used to be one decision you made once. Now the cost lever lives inside your code, and the same application can send its urgent calls on Priority and its patient ones on Flex.

The moment you stop paying per token and start reserving capacity, a new rule takes over. Your bill is set by how full that capacity is, not by how much work you sent through it.

SageMaker offers four inference modes: real-time, serverless, asynchronous, and batch transform. Real-time endpoints stay on and bill you for every hour, busy or idle. The trap is utilization. An endpoint running around the clock against variable traffic with no autoscaling commonly wastes somewhere between 40 and 65 percent of its spend on hours where it served almost nothing.

There is a detail here that catches experienced teams off guard. Managed Spot pricing, the deep discount many of us reach for first, applies to SageMaker training jobs and not to inference endpoints. The lever you expect is not on this door.

Provisioned Throughput on Bedrock obeys the same logic from the other direction. A Model Unit guarantees throughput and bills hourly regardless of use, which is excellent at steady high volume and punishing at anything sporadic.

There is also a middle door people forget exists. Bedrock Custom Model Import lets you bring your own trained weights, for architectures like Llama and Mistral, and run them in the managed serverless pattern without standing up infrastructure yourself. The choice was never only “managed model from the provider” or “host everything yourself.”

This is the path with the lowest price per token and the deepest trap, so it earns the most attention.

Running open weights on your own EC2 or EKS capacity, especially on Spot instances, can cut raw inference cost by 60 to 70 percent against managed on-demand. That number is real. It is also conditional, and the condition is utilization. A GPU you rented and left half empty is more expensive per useful token than the managed API you were trying to beat. Whether that GPU stays full is decided almost entirely by the serving engine, so it is worth understanding what the good ones do.

Start with batching, the way a GPU serves many requests at once. The naive method fills a batch, starts it, and waits for every request in it to finish before the next batch begins. The catch is that requests finish at different times. A short answer is ready in a second, but it stays locked in its slot, producing nothing, until the longest answer in the batch catches up. The GPU spends most of its time waiting. You bought a Ferrari and you are driving it in first gear.

The second technique is PagedAttention, the one that made vLLM famous. To see what it does, you need one fact about how these models run. Every active request needs memory to remember the tokens it has already processed, and that memory is called the key-value cache, or KV cache. The naive approach reserves one large block of it per request, sized for the longest answer that request might produce. Most answers come out shorter, so most of that reserved memory sits empty. Older systems wasted 60 to 80 percent of it this way.

PagedAttention borrows a trick from operating systems called paging. Instead of one large block, it cuts the cache into small fixed pages, sixteen tokens each, and hands them to a request only as it actually needs them. The pages do not have to sit next to each other in memory. Waste falls to under 4 percent, and all the memory you get back becomes room for more requests running on the same GPU at once.

Once you see the three operating models clearly, the real decision stops being “which has the lowest price per token” and becomes a small set of questions that a price sheet cannot answer.

What does your utilization actually look like over a full week? Steady, high, predictable volume bends toward dedicated capacity or self-hosting. Bursty and unpredictable traffic bends toward managed pay-per-token, because you stop paying the moment the burst ends.

What is your latency target, and what shape does it have under load? Median latency is easy to hit on a quiet afternoon. The tail, the slow one percent that your worst-case user feels during a spike, is where capacity decisions get expensive.

Who carries the pager when the serving layer falls over at three in the morning? On the managed path that is the provider. On the self-hosted path that is your team, and that on-call cost is real money that never shows up on a per-token comparison.

Every pricing page in this space leads with price per million tokens. None of them lead with your utilization, and utilization is the multiplier that decides what you actually pay. The headline number is the one input you control the least and study the most.

The service tiers and the discounts read as universal in the announcements. In practice they arrive model by model and region by region. The combination you want may not exist yet in the region you need, and that is a design constraint, not a footnote.

On the dedicated path, the meter runs during silence. An idle reserved endpoint is not a small inefficiency to clean up later. It is the whole problem, and the standard reflex of reaching for Spot pricing does not apply to inference the way it does to training.

On the self-hosted path, the attractive price per token is a ceiling you only reach with a serving engine doing continuous batching and paged memory well. Quote that number without the serving work behind it and you are quoting a result you have not earned yet.

I have spent the last several weeks in this series writing about the layer above this one. The agents, the Model Context Protocol servers, the multi-agent setups on AgentCore. Every one of those designs quietly assumes the serving layer as a solved input. The agent calls an endpoint and trusts that tokens come back.

That assumption gets expensive fast. An agentic workflow turns one user action into a chain of model calls, sometimes ten or more, before the user sees a single word. The open-source assistant harnesses getting attention right now make the point cleanly, because they are explicitly bring-your-own-model. The harness hands you the agent and hands you the serving bill at the same moment. The architecture you drew for the agent last quarter is the thing that set your inference spend this quarter, and most teams never connect the two diagrams.

Hold three sentences in your head and most of this decision makes itself. Managed inference pays per token and costs you nothing while it sits idle. With dedicated capacity you pay for the reservation, and every idle hour is money gone. Self-hosted serving pays for raw GPUs, and whether that turns out cheap or ruinous comes down to the scheduler underneath.

Price per token only describes the first sentence. Before you move a workload to save money, measure your real utilization across a full week and name the person who gets paged when it breaks. If you cannot answer both, you are not comparing operating models. You are comparing brochures.

This is part of Golden Jacket Field Notes, a weekly practitioner series on production Machine Learning, Generative AI, and cloud architecture on Amazon Web Services.

Erick Mancz is an AWS Solutions Architect and Golden Jacket holder focused on production ML and GenAI workloads for regulated and high-stakes organizations. He holds every active AWS certification, including the Generative AI Developer Professional as an Early Adopter, and co-founded Golden Jackets Brazil. He writes Field Notes from workloads where outages, audit gaps, and cost surprises are not tolerated.

Medium: medium.com/@erickmancz · LinkedIn: linkedin.com/in/erick-mancz · AWS Builder Center: builder.aws.com/community/@imancz · Credly: credly.com/users/erick-mancz.66783c3e

Opinions expressed here are my own and do not represent the views of my employer or any affiliated organization.