
Own Your Inference: The Token Economics of Open-Weight Models on AWS
By Shrisha Kulkarni, Dhruv Yadav, Sahasra Challa, Kevin Wang
Series: open-weight-models (1 article)
- 1Own Your Inference: The Token Economics of Open-Weight Models on AWS This article
A decision framework for teams choosing between frontier APIs and self-hosted open-weight models - with the arithmetic to back it up.
The model that vanished
On June 12, 2026, a US government export-control directive forced Anthropic to disable Claude Fable 5 and Mythos 5 for every customer in the world. There was no deprecation window and no migration guide. The models were simply gone. Access wasn't restored until July 1.
This isn't a criticism of Anthropic. They complied with a lawful directive and communicated it clearly. But if your product's behavior was pinned to one of those models, you spent three weeks learning an uncomfortable lesson about a dependency you didn't know you had:
When a model lives entirely behind someone else's API, its availability is subject to forces that neither you nor even the vendor fully control.
Open-weight self-hosting is not inherently cheaper. It becomes cheaper only under a narrow and measurable set of conditions. This article gives you both halves: the reasons open weights matter, and the arithmetic that tells you whether they'll actually save you money.
To keep the arithmetic concrete, we'll return throughout to one industry where these pressures are sharpest: financial services. Banks, trading desks, and insurers combine strict regulation (FINRA/SEC audit trails, model explainability, cross-border data rules) with enormous, repetitive inference volumes (fraud scoring, document processing, client communications). That combination is exactly where the open-weight case stops being theoretical.
1. The premise: default != optimal
Frontier APIs are the right default for most workloads, most of the time. Pay-per-token pricing costs nothing at rest, scales instantly, and requires no operational investment. If you're not sure what to do, use one.
But "default" and "optimal" aren't the same thing. Three workload profiles are where a custom open-weight model on AWS delivers better cost-efficiency and customization than a general-purpose frontier API:
- High-volume inference - narrow, repetitive tasks at scale, where per-token cost compounds into a serious line item.
- Domain-specific tasks - where a small, fine-tuned model matches frontier quality on your one task, at a fraction of the cost.
- Strict data-residency environments - regulated teams whose data legally cannot leave their Region.
If your workload looks like one of these, read on. If it doesn't, a managed frontier API is probably still your best bet.
2. Why open-weight matters: compliance, continuity, and control
Cost is one of the loudest argument for open weights, but it's rarely the one that forces the decision. More often it's a requirement that leaves you no choice.
Data residency and sovereignty. For workloads under GDPR, or under public-sector and sovereign-cloud mandates, where your data is processed is a hard requirement, not a preference. An open-weight model running in a specific AWS Region keeps your prompts and completions inside a boundary you can point to during an audit. A pay-per-token API may route requests through regions or accounts you don't control - and "trust us" is not a compliance posture.
Regulated data, like PHI under HIPAA. When prompts contain protected health information, the question becomes: is this endpoint in scope for my Business Associate Addendum, and can I prove the data never left my controlled environment? Self-hosting on AWS lets you run inference inside a HIPAA-eligible architecture you own end-to-end - your logging, your retention, your access controls. Vendors now offer strong guarantees such as Zero Data Retention; self-hosting is how you provide those guarantees yourself, on your own terms.
Continuity and deprecation risk. This is the Fable 5 lesson generalized. Proprietary models get deprecated on the vendor's schedule, not yours - and sometimes, as June showed, on no schedule at all. If your application is validated against a specific model version, a forced migration means re-testing, re-validation, and sometimes re-certification. An open-weight model you've downloaded and deployed stays exactly as it is until you decide to change it. For a financial-services firm, that re-certification isn't a formality - a fraud-detection or trading-support system pinned to a deprecated model can mean frozen operations and a compliance re-review, with no fallback the day the model vanishes.
Data and IP control. With an open-weight model on your own infrastructure, your prompts, your fine-tuning data, and your model weights never become telemetry or training signal for someone else. For teams whose prompts are the product, that's decisive.
These four reasons are qualitative. They can justify self-hosting even when the math says the API is cheaper - but you should know how much you're paying for them. That's what the rest of this article is for.
3. Token economics: the shape of the decision
You don't need a spreadsheet to grasp the core idea. The two options simply have different cost shapes.
A pay-per-token API costs you nothing at rest and scales up with every request. Self-hosting is the mirror image: a mostly fixed monthly cost for the instance, and then inference that's effectively free up to that instance's capacity. The more you use it, the cheaper each request gets.
Those two lines cross. Below a certain steady volume the API is cheaper. Above it, self-hosting wins, and the margin keeps widening as volume grows.
The whole question is: where does the crossover sit for your workload?
For our financial-services example, the answer usually tips toward self-hosting: fraud scoring and document processing run steadily, all day, at high volume - the kind of sustained load that pushes a firm well past the crossover and makes the fixed-cost model the cheaper one.
3.1 The four deployment modes
Framing this as a binary - API vs. self-host is a mistake. There are four modes, and the middle two are where a lot of teams should actually land:
The real question is not "should I self-host?" It's "which of these four modes fits my utilization curve, my quality bar, and my operational maturity?"
4. The three-condition threshold
Here is the single most important finding in this analysis, and the one that most enthusiasm for open weights skips past.
Test it against our running example. Three conditions must hold simultaneously for open-weight deployment to deliver cost savings:
- Sustained GPU utilization above roughly 30%. Below this, the fixed cost of dedicated hardware exceeds what variable per-token pricing would cost for the same workload.
- Genuine quality parity on the target task - not merely acceptable performance on general benchmarks. A smaller or specialized open-weight model has to actually do the job.
- Operational maturity to run inference systems - or willingness to absorb the managed-service premium from a third-party host.
When any one condition fails, the economics collapse in favor of API consumption.
This is why the instinct to self-host "to save money" is, in the general case, economically unfounded below a few hundred million tokens per month. It's also why the picture changes dramatically for narrow, fine-tuned tasks - as we'll see in section 7.
5. What to measure
The three conditions are only checkable if you instrument your pipeline. These are the parameters an inference pipeline should log continuously - and the numbers you need before anyone builds a business case.
Volume metrics
Performance metrics
Quality metrics
Two measurement traps to avoid:
- Benchmark at real concurrency, not single-request.
- Cost per unit of work beats monthly total. Track cost per classification, per document, per conversation. The monthly total tells you nothing about whether you're winning.
6. The break-even math, worked
Let's do the arithmetic that most posts hand-wave.
6.1 The self-hosted side
Representative AWS GPU instances (us-east-1, on-demand, July 2026):
Critically, raw instance cost is not the real cost. A production stack carries ongoing operational load - on-call, monitoring, upgrades, autoscaling - that an hourly rate omits. Carry it, and that g5.2xlarge running a 7-9B model is closer to $2,561/month than $883. Every honest comparison uses the loaded number.
Spot ($0.939/hr) and 1-year RIs ($0.764/hr) cut the hardware line by 22% and 37%. But operational load is people, not silicon, so it does not shrink: against the loaded figure those commitments save only ~8% and ~13%, and on an RI overhead is three-quarters of total cost.
6.2 The break-even points
Using that g5.2xlarge configuration running a 7-9B parameter model:
The Fable 5 comparison looks like a slam dunk - until you notice it's theoretical rather than practical: no open-weight model at that price point matches Fable 5's reasoning capability. You cannot substitute what you cannot match. Condition 2 fails.
6.3 The utilization problem
Here's the number that should reframe your intuition. That same g5.2xlarge can deliver approximately 16 billion output tokens per month at full concurrency:
Break-even against Haiku 4.5 requires only ~500M of those tokens - about 3% of theoretical capacity.
So why doesn't self-hosting obviously dominate everywhere? Because real production workloads run at 5-30% utilization, not 100%. The gap between theoretical capacity and actual usage is precisely where the economics of self-hosting most commonly fail. You are not billed for the tokens you could have generated.
6.4 Provider selection is a first-order lever
For identical open-weight models, third-party inference providers vary by up to 19x on price and 7x on throughput. (DeepSeek V4 Pro ranges from $0.04/task first-party to $0.76/task on Fireworks.)
The implication is sharp: choosing the wrong provider for open-weight inference can entirely negate the cost advantage of choosing open weights over a frontier API. Provider selection is not a secondary optimization - it is a primary economic lever of the same magnitude as the self-host-versus-API decision itself.
6.5 How to find your own crossover
- Measure sustained throughput (req/s and tokens/s) under real concurrency.
- Measure the utilization fraction you actually achieve. Idle GPU time is pure cost.
- Measure your input-to-output token ratio. Output is the expensive side and drives the API bill.
- Establish steady requests/day.
- Divide the instance's hourly cost by its sustained throughput -> a self-hosted cost per token you can compare directly against the API's published rate.
- Amortize any fine-tuning cost across expected lifetime volume (section 7).
- Use the loaded instance cost, not the raw one.
7. Moving the crossover in your favor
The crossover point isn't fixed. Two families of levers move it.
7.1 Spend fewer tokens per request
- Trim the prompt. Strip boilerplate and redundant instructions from system prompts; every token is paid on every call.
- Cache the fixed prefix. Prompt caching stops you reprocessing the same system prompt, few-shot examples, or retrieved context on every request. Note that "blended price" figures in public leaderboards typically assume a 7:2:1 cache-hit:input:output ratio - heavily weighted toward cache hits, which naive $/token comparisons ignore.
- Cut the output. Output tokens are generated one at a time and cost more than input tokens, so verbose completions are where bills balloon. Constrained or structured outputs and hard max-token caps target the expensive half of the bill directly.
7.2 Get more work out of each instance
- Continuous batching (vLLM / TGI). Interleave many requests through the GPU at once. Often the single biggest throughput gain - reported multiples range from ~8x for batching alone up to ~23x versus a static baseline.
- Quantization (FP8 / INT8). Fit a cheaper GPU and run faster, usually near-lossless for narrow tasks. Published FP8 results on Mistral 7B (H100, batch 32) show ~33% higher tokens/sec, ~24% cost reduction, and ~8.5% lower TTFT at near-identical perplexity versus FP16.
- Right-sizing and autoscaling. Don't pay for idle capacity between traffic peaks. This is the direct attack on the utilization problem in 6.3.
7.3 Fine-tuning: the lever that actually changes the answer
This is where the two halves of the open-weight argument meet. Open weights don't just save money at scale - they let you build something a shared API can't give you: a model tuned to your domain. And a well-tuned small model is what makes condition 2 (quality parity) achievable.
The evidence is strong. Predibase's LoRA Land work fine-tuned 25 models on a single A100 at roughly $8 per fine-tune, and the resulting adapters beat GPT-4 on 25 of 27 specialized tasks - with margins like NER 0.99 vs 0.75 and dbpedia 0.99 vs 0.04.
That reframes the economics entirely. The strongest case for open-weight deployment is not general-purpose substitution - it's task-specific specialization at sustained volume. For high-volume classification, extraction, or structured-output work, fine-tuning plus self-hosting is where the math turns decisively favorable, and at volumes far below the 500M-token general-purpose threshold.
8. Fine-tuning on AWS: a practical guide
Fine-tuning is the process of taking a pre-trained model and continuing to train it on a smaller, curated dataset so that it specializes in a particular task, domain, tone, or output format. The base model already carries broad general knowledge; fine-tuning doesn't replace that knowledge so much as it reshapes behavior toward a narrower objective.
One clarification is worth stating at the outset. Fine-tuning is not a general-purpose mechanism for improving model quality. It is well suited to enforcing a consistent output format, adopting a tone or persona, teaching domain vocabulary, or producing a smaller and cheaper model that performs competitively on one narrow task. It is poorly suited to supplying factual knowledge that changes frequently, which is more appropriately addressed through retrieval-augmented generation (RAG).
A practical diagnostic: does the desired output change when a source document changes (-> retrieval), or when the task itself changes (-> fine-tuning)? Distinguishing these two cases is the first and most important decision in any fine-tuning effort.
8.1 Techniques, lightest to heaviest
Start with the lightest approach that could plausibly solve the problem; adopt heavier methods only when a lighter one proves insufficient.
- Prompt engineering and RAG (before any fine-tuning). Confirm the problem can't be solved here first.
- Parameter-efficient fine-tuning (LoRA / QLoRA). Freezes the original weights and trains a small set of additional "adapter" parameters. Substantially less memory and time than full fine-tuning, produces a small artifact that can be stored and swapped inexpensively (a 40 GB model yields adapters of a few MB, ~0.19% trainable parameters), and achieves quality close to full fine-tuning for most specialization tasks. QLoRA additionally compresses the base model before training, letting 40-70B models be tuned on modest hardware. For the majority of use cases this is both the recommended starting point and the endpoint.
- Supervised fine-tuning (SFT). The baseline instruction-tuning objective - training on input/output pairs to teach format, tone, and domain behavior. The parameter-efficient methods above are typically implementations of this objective.
- Preference tuning (DPO / RLHF). For when output quality is a matter of judgment rather than objective correctness. DPO trains on pairs of preferred and rejected responses and is considerably simpler and cheaper than classical RLHF.
- Continued pre-training (domain adaptation). Training further on large volumes of unlabeled domain text so the model absorbs the vocabulary and patterns of a field before task-specific tuning. Also covers adapting a model to a new language.
- Full fine-tuning. Every weight updated. Highest ceiling for deep specialization, but the most memory, the longest runtime, and a full-size artifact per version. For small models, rarely necessary.
8.2 The journey
The process is the same regardless of technique, and it's iterative, not a single terminal effort:
- Define the gap. Articulate precisely what the model does incorrectly and what a corrected output looks like.
- Select a base model, balancing size, cost, and license terms. A smaller, well-tuned model frequently outperforms a larger general-purpose model on a narrow task at a fraction of the serving cost. Larger is not automatically preferable. Review and accept license terms at this stage.
- Curate the dataset. This is where most of the effort and most of the resulting quality reside.
- Pick a technique, starting light, and run a small pilot to surface data and formatting errors before committing to a long run.
- Evaluate against the un-tuned base on a held-out set with task-specific metrics.
- Deploy and monitor for drift, re-tuning on a cadence as data and requirements evolve.
The central point: the model is rarely the limiting factor - the dataset is. Organizations that invest in data quality and rigorous evaluation tend to succeed; those that treat fine-tuning as a single-step quality improvement are frequently disappointed.
8.3 Dataset requirements
Format. Training data is typically JSON Lines with one example per line (SageMaker also accepts JSON, compressed JSON, and Arrow). For SFT each line is a prompt-completion pair; for preference tuning, a prompt plus a chosen and a rejected response. Always provide a validation split - or permit the service to reserve one - so overfitting can be detected. The dataset must conform to the base model's expected chat format and special tokens; a mismatch here is a common and silent source of failure, where training appears to proceed correctly but the deployed model behaves anomalously.
Quality. A few principles account for most of the outcome:
- Quality over quantity. A few hundred to a few thousand clean, consistent, representative examples typically outperform tens of thousands of noisy ones.
- Consistency is essential. The model learns whatever patterns are present - including your errors and internal contradictions.
- Representativeness matters. Cover the full range of production inputs, including edge cases and failure paths.
- Cleanliness and safety. Deduplicate, remove PII that shouldn't be memorized, and filter off-brand or harmful content unless that content is the explicit object of the task.
- No leakage. Evaluation examples must not appear in the training set, or your metrics are meaningless.
A compliance note: because fine-tuning occurs within your own AWS account, the training data never leaves your Region and is never held by a third party - a decisive consideration for protected health information, legal documents, and trade secrets.
8.4 Evaluation
A fine-tuned model should never be deployed on the basis of informal impressions. Evaluate across three dimensions: task quality, safety and responsibility, and regression relative to the base.
- Automatic signals during training. Monitor loss and perplexity. A training loss that continues to decline while validation loss plateaus or rises indicates overfitting.
- Task-appropriate metrics. ROUGE for summarization, BLEU for translation, exact-match or F1 for extraction and QA, pass@k for code generation, accuracy for classification.
- Managed tooling. SageMaker Clarify's Foundation Model Evaluations (FMEval) assess accuracy, robustness, toxicity, and bias - via a guided workflow in Studio or an open-source library that accommodates custom datasets - and results map to responsible-AI frameworks including ISO 42001 and the NIST AI Risk Management Framework. For agentic fine-tuning, SageMaker's model-customization evaluation reports reward, pass@k, and trajectory metrics relative to the base. Track runs in managed MLflow so the successful configuration can be reproduced.
- Human or model-as-judge review where quantitative metrics fail to capture tone, helpfulness, and brand alignment - with judge models spot-checked against human ratings.
The one requirement that cannot be omitted is a direct comparison against the base model on data the model did not encounter during training. Fine-tuning can induce catastrophic forgetting, where the model improves on the target task while its general capability degrades. A general-capability check must therefore accompany the task-specific metric. A model that does not surpass the base on the target task without impairing general ability should not be deployed.
8.5 The four AWS paths
Ranging from most fully managed to most fully controlled, corresponding to how much of the stack you want to own:
- Amazon Bedrock, managed fine-tuning. For supported models, Bedrock handles both customization and serving, and now exposes OpenAI-compatible fine-tuning endpoints so familiar tooling works. Least infrastructure effort.
- Bedrock Custom Model Import. Fine-tune anywhere, then import the weights into Bedrock for on-demand serving. Scales to zero when idle - no cost for maintaining a warm instance - which suits spiky or low-volume workloads. Data stays in-Region, artifacts transfer over a VPC endpoint, and customer-managed KMS keys are supported.
- Amazon SageMaker AI. The most flexible option. JumpStart fine-tunes more than twenty popular open models with minimal code; training jobs give full control on pay-per-second clusters that are provisioned and released automatically; HyperPod supports large-scale, distributed, resilient runs through ready-made recipes. Supports SFT, DPO, and RL methods on either GPU or Trainium.
- Self-managed on EC2, EKS, or ECS. Run your own training stack on GPU instances (P5, G5/G6) or Trainium (Trn1, Trn2). Maximum control at the cost of maximum operational responsibility; suits teams with strong infrastructure capabilities or specialized requirements such as air-gapped or GovCloud environments.
A common thread across all four: access to hardware unavailable through model-provider APIs - Trainium for cost-efficient training, Inferentia for economical serving, and Graviton for CPU-side work. Published comparisons indicate Trainium is frequently less expensive per training-hour than equivalent GPUs.
8.6 What fine-tuning costs
At its core: instance hourly rate x number of instances x training hours, plus supporting costs. Unlike proprietary per-token fine-tuning, every input to this calculation is visible and controllable.
The practical method for estimating training hours is to run a small pilot, measure its duration, then scale by full dataset size and epoch count. Estimate large runs by measurement and extrapolation, not analytically.
Factors that govern the total: model size (larger models need more expensive instances and run more slowly); technique (LoRA/QLoRA are frequently 5-10x cheaper than full fine-tuning because they fit on a single smaller instance rather than a cluster); dataset size and epochs; sequence length (longer inputs cost disproportionately more); and hardware utilization (Spot capacity and correct instance sizing routinely cut 40-60%).
Order of magnitude, as of mid-2026 in us-east-1:
Don't overlook the surrounding costs: S3 storage for datasets and checkpoints, the several trial runs that typically precede a definitive one, evaluation compute, and serving - usually the larger cost over the model's lifetime. Always account for one-time training cost separately from ongoing inference cost, and use the AWS Pricing Calculator for authoritative figures.
8.7 Sunk cost or amortized asset?
Fine-tuning should be regarded as an amortized investment, not a sunk cost. You pay once for the training run and data preparation, then realize the benefit across every inference for the operational life of the model.
The right mental model isn't "$X to fine-tune" - it's that one-time cost spread across the total volume of requests the model will serve. A $2,000 training run that subsequently serves 50 million requests contributes a fraction of a cent per request, negligible relative to inference cost. As usage increases, the per-request contribution of training cost approaches zero.
This is the inverse of proprietary per-token pricing, which scales linearly with volume and can be raised by the provider at any time. A model you own behaves the opposite way: training cost is fixed and its per-request share declines as volume grows, while inference cost per request itself falls as the deployment is right-sized, quantized, batched, and migrated to Inferentia or Trainium.
The cost behaves as a genuine sunk cost in only two circumstances:
- A run that fails evaluation and doesn't improve on the base - which is precisely why cheap pilot runs and a strict evaluation gate matter. They keep failed experiments cheap.
- A base-model upgrade that retires an existing fine-tune - though this happens on your schedule rather than a vendor's, and a retained dataset plus a reproducible recipe makes re-tuning on a new base inexpensive. Your investment in data amortizes across successive model generations, not just a single model.
Budget fine-tuning as an amortized asset rather than a one-time gamble. Keep experiments cheap through parameter-efficient methods and gate them with rigorous evaluation, so successful runs repay their cost many times over as volume grows while unsuccessful runs stay inexpensive.
9. The July 2026 market inflection
Any analysis of this space is time-bound, and the April-July 2026 window represented a structural shift worth understanding.
Anthropic cannibalized its own top tier. The release of Opus 5 - at half the price of Fable 5 with near-equivalent quality, $5/$25 per M tokens - substantially devalued the "self-host to escape frontier pricing" argument at the top end.
Open weights reached near-frontier capability. Models from Moonshot (Kimi K3) and Zhipu (GLM-5.2) achieved near-frontier capability scores while remaining available through serverless providers at a fraction of frontier API pricing - Kimi K3 at roughly $3/$15 per M.
The consequence is that the previously relevant comparison - frontier API versus self-hosted open weights - has been supplanted by a new decision axis: Opus 5 versus Kimi K3 via managed serverless hosting. For frontier reasoning and agentic workloads, no economically viable self-hosting path exists today. The practical choice at that tier is between closed API providers and near-frontier open weights on managed serverless.
A regulatory overlay applies. Several of the most cost-effective open-weight models (Kimi K3, GLM-5.2, DeepSeek V4) originate from Chinese laboratories. Active policy discussion about restricting these models in US commercial and government contexts introduces a supply-continuity risk that organizations should hedge against when making commitments beyond twelve months. Note the irony: this is the same class of risk that motivated the opening story.
10. An existence proof at the extreme
Lest the skeptical analysis above read as an argument against ever self-hosting: Character.AI operates at roughly 20,000 QPS - about 20% of Google Search volume - at under $0.01 per hour of conversation, reporting 13.5x savings versus commercial APIs and a 33x cost reduction since 2022, achieved through aggressive KV-cache and inter-turn caching techniques.
At sufficient scale with sufficient engineering investment, self-hosting wins by more than an order of magnitude. The question was never whether the crossover exists. It's whether your workload sits past it - and whether you have the operational maturity to collect the win.
The counterweight, equally instructive: Sourcegraph's published analysis of running code intelligence in-house cites 3-year TCO ranging from $5.5M to $58M, with ~$1.1M/year in maintenance and substantial hidden operational costs (fork drift, permissions, plumbing). That range is the operational-maturity condition, quantified.
11. Recommendations by workload pattern
The operational principle: default to API-based inference and migrate to dedicated infrastructure only when empirical utilization data demonstrates the break-even threshold has been crossed. Measure first. Migrate second.
12. Why AWS
Everything above converges on a single environment.
- Your data stays in-Region. A residency-controlled, HIPAA-eligible architecture you own end-to-end.
- One stack, one bill. The compliance boundary, the data controls, the fine-tuning loop, and the inference all live in the account where your data already is.
- The full spectrum. Amazon Bedrock for managed, serverless serving of supported open models (including NVIDIA Nemotron 3 Nano); Amazon SageMaker AI for full control of training and fine-tuning; EC2 with Trainium, Inferentia, and Spot capacity when you want to own the economics down to the instance.
- You own the economics and the continuity. No third-party directive can switch your model off.
And here's the part the inference specialists can't match: on AWS you don't have to choose. You can run frontier APIs and self-hosted open-weight models side by side in the same stack - frontier for the hard, spiky, low-volume reasoning, open-weight for the high-volume, domain-specific, residency-bound work. Route each workload to whichever option wins, instead of betting the whole company on one.
You're not renting an endpoint. You're owning your inference.
13. Open questions and forward view
This analysis is necessarily time-bound. The pricing data and model capabilities referenced are current as of July 2026, and the pace of change in this market renders any fixed analysis stale within one to two quarters. The framework - utilization, quality parity, operational maturity - is durable. The specific thresholds must be recalculated as hardware costs decline and API pricing evolves.
Next-generation hardware. NVIDIA's Vera Rubin platform is projected to deliver roughly 5x performance-per-dollar improvement over current-generation hardware by 2027 (Semianalysis analyzes it at $3.57 per GPU-hour operator TCO, against $1.84 for GB200 and $2.36 for GB300). This will lower self-hosting break-even thresholds - but API providers will capture some of the same hardware efficiencies. The net effect is likely a gradual expansion of the workload envelope where self-hosting wins, not a sudden inversion of current economics.
Benchmark literacy. Two caveats that trip up a lot of comparisons. Composite indices like Artificial Analysis's Intelligence Index aggregate nine evaluations - a model scoring 60 performed well across a specific weighted mix of tests, not on any single benchmark. And "cost per task" on those leaderboards is not a per-token rate; reasoning models generate more tokens per task and therefore show higher cost per task even at identical per-token prices.
Tokenizer differences matter. Claude 4.7+ and Mythos Preview use a new tokenizer producing roughly 30% more tokens for the same English text (measured multipliers: 1.42x English, 1.33x Spanish, 1.28x Python, 1.01x Mandarin). Compared prices between model generations are not apples-to-apples on a per-word basis.
14. Conclusion
The economics of open-weight LLM deployment are knowable, measurable, and - for most organizations today - unfavorable relative to API consumption on general-purpose workloads. The framework reduces the decision to arithmetic: measure utilization, validate quality parity, account for operational cost, compare against current per-token pricing.
But that's only half the picture, and the half that most cost analyses stop at. Two things shift the answer:
Specialization. A fine-tuned small model on a narrow task isn't competing with a frontier model on general reasoning - it's beating it on the one thing you need, at a fraction of the cost. That's where the crossover arrives early and the returns compound.
Requirements that aren't about money at all. Residency, PHI, IP control, and continuity can make self-hosting the only option regardless of what the spreadsheet says. When that's true, the arithmetic still matters - it tells you the price of the guarantee you're buying.
When the math favors migration, migrate. Until then, the API remains the rational default. Either way, measure before you build the business case - and know which of the two arguments, cost or control, is actually driving your decision.
Try it yourself
The best way to internalize where the line sits for your workload is to build it. Deploy a small open model on AWS, apply the token-optimization levers and watch the economics shift, fine-tune it with LoRA, and walk away with a defensible answer to "is open-weight right for us?" - one you can take to both your CFO and your compliance team.
Series: open-weight-models (1 article)
- 1Own Your Inference: The Token Economics of Open-Weight Models on AWS This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article