Fixing the Token Economy: A Cost-Aware LLM Router, Proven on Amazon SageMaker AI and Amazon Bedrock
Route the easy 74% of your LLM traffic to a self-hosted open model, reserve frontier models for the requests that actually need them — built with Amazon Bedrock AgentCore Gateway, Amazon SageMaker AI, and Amazon Bedrock.
The challenge: token economics
Every request to a frontier model costs the same premium price whether the task is "reverse a string" or "design a distributed rate limiter." The token bill grows with usage, not with task difficulty, and most pricing models give you no dial for "this one didn't need your best model." As LLM-powered features move from pilot to default-on across more of the developer workflow, that gap between what a request costs and what it actually needed compounds fast — it's the single most common reason a promising AI feature's unit economics stop working the moment it succeeds at scale.
What we're seeing in the field
Three patterns show up consistently once you look at real production LLM traffic instead of assumed traffic:
- Most requests are routine. In our own measured traffic, roughly three-quarters of coding-assistant-style requests were the kind a well-tuned smaller model handles correctly on the first try — bug fixes, test scaffolding, docstrings, small refactors. The genuinely hard fraction — architecture design, ambiguous multi-system tradeoffs — is real, but it's a minority.
- Teams default to the expensive model for everything, not because the easy majority needs it, but because building a reliable way to tell the two apart — and a safety net for when that classifier is wrong — is its own engineering project, not a config flag.
- The cost conversation usually stays theoretical. "We could save money by routing to a cheaper model" is a common slide; a measured offload ratio, a measured quality-parity check, and a real infrastructure cost model attached to it is much rarer. The gap between the idea and a working, measured version of it is exactly where most attempts stall.
The solution
A cost-aware router closes that gap directly: classify each request, send the routine majority to a self-hosted open model, reserve the frontier model for the requests that actually need it, and add a safety net that catches the rare bad answer before a caller ever sees it. The idea is simple. What follows is a working version of it, built entirely on AWS, with real measured numbers behind every claim below — not an estimate.
What we built
A single, OpenAI-compatible endpoint that looks like one model to any calling application, but is actually two:
- A self-hosted 27B open model (
Qwen3.8-27B-AWQ-INT4) running on Amazon SageMaker AI, handling the majority of requests at a fraction of frontier per-token cost. - Claude on Amazon Bedrock, reserved for requests a lightweight classifier flags as genuinely complex, plus a safety-net path that rescues any Qwen answer that looks wrong before it ever reaches the caller.
The whole thing is fronted by Amazon Bedrock AgentCore Gateway, which gives both backends one IAM identity and one place to enforce auth — so the routing layer itself never needs to hold or manage AWS credentials.
How the routing decision works
Every request passes through the vLLM Semantic Router — an open-source Envoy
ext_proc layer — before any model is called. Two signals decide the outcome: a domain match (does this look like a coding/technical request at all) and a complexity score against a calibrated threshold. Below the threshold, the request goes to the self-hosted model first. Above it, straight to Claude.
The threshold itself is worth being precise about:
0.18 isn't a default or a guess, it's calibrated against real measured traffic — easy prompts in our data scored −0.079 to 0.098 on this signal, genuinely hard ones scored 0.242–0.288, and the threshold sits in the gap between them. That's the difference between a routing rule that looks reasonable and one that's actually been checked against production behavior.There's a second, reactive safety net behind that: a lightweight verifier checks every self-hosted-model answer after the fact — did it refuse unexpectedly, get cut off mid-answer, produce invalid syntax, or come from a low-confidence classification — and automatically rescues anything suspicious via Claude before the caller ever sees it. The two paths mean cost savings never come at the expense of a silent quality failure; the worst case is paying for both calls on the rare answer that needed rescuing, not shipping a bad one.
Why this architecture
Why Qwen3.8-27B. The self-hosted tier only makes sense if the open model is actually good, not just cheap. On LiveCodeBench v6, Qwen3.8-27B scores 90.3 — ahead of Anthropic's own Opus4.6 Max reference point at 88.8 on the same benchmark, and it holds up on real software-engineering tasks too (61.7 on SWE-bench Pro, 79.0 on QwenSWEBench). This isn't a fluke or an isolated case: open-weight models have closed most of the gap with frontier models on coding specifically, faster than almost any other capability area, because coding has the clearest, most verifiable training signal of any task class — you can check whether generated code actually runs and passes tests, which makes it unusually tractable to train and benchmark rigorously. That's the real story behind why this architecture works economically: the cheap tier isn't a compromise on quality for the routine majority of requests, it's a genuinely competitive model for the coding domain specifically. (One honest caveat: the benchmark numbers above are for the base BF16 model — AWQ-INT4 quantization, what we actually run in production, isn't separately benchmarked on this card, though the shift is typically small at this precision.)
Why Amazon Bedrock AgentCore Gateway. Routing between a self-hosted model and a managed frontier model normally means building and maintaining two different auth and request-shaping paths. AgentCore Gateway collapses that into one: a single execution role holds both
sagemaker:InvokeEndpoint and bedrock:InvokeModel permissions, passthrough targets forward requests to either backend with zero server-side translation, and guardrail/policy checks sit at the Gateway layer instead of being reimplemented per backend. Repointing production traffic to a new model version — something we did repeatedly while tuning this — is a single API call, not a redeploy.Why Amazon SageMaker AI. Self-hosting the cheap tier only pays off if you have real control over the serving stack: the exact model weights, the chat template, the quantization scheme, the instance type. SageMaker AI's real-time inference endpoints give you that control directly, while still handling the operational parts (health checks, IAM-scoped invocation, CloudWatch metrics) you don't want to build yourself.
Why Amazon Bedrock. The frontier tier needs to be zero-ops: no capacity planning, no version management, and access to multiple model families (we route between Claude Sonnet and Claude Opus depending on task difficulty) through one consistent API. Bedrock is the tier you want to think about the least — which is exactly the point, since it's reserved for the smallest share of traffic.
Put together: Qwen closes the quality gap enough to make self-hosting worth doing at all, SageMaker AI gives you control where control saves money, Bedrock gives you zero-ops access where reliability matters more than cost, and AgentCore Gateway is what makes routing between the two a configuration decision instead of an integration project.
Results
Measured against live production traffic, not a benchmark environment:
- 74.2% offload ratio — nearly three-quarters of real requests served entirely by the self-hosted model.
- 9.7% escalation rate — the share of requests where the safety-net path stepped in, down from 25.8% before tuning.
- ~35 tok/s sustained per-request throughput, scaling to 212.6 tok/s aggregate at 16 concurrent requests through continuous batching on a single GPU instance — the self-hosted tier scales with concurrent load, not just raw traffic volume.
The economics. At full GPU utilization, self-hosted generation costs roughly $2.93 per million tokens — about 3.4x cheaper than Claude Sonnet's output rate, and 8.5x cheaper than Claude Opus's:
Here's what that turns into at different traffic volumes, computed from measured throughput and published API pricing — not a vendor estimate:
| Scenario | Requests/month | Frontier cost avoided | GPU infra cost | Net savings | Annualized |
|---|---|---|---|---|---|
| 500 developers, light usage | 750K | $8,370/mo | $1,614/mo | $6,760/mo | $81K/yr |
| 500 developers, heavy usage | 2.25M | $25,110/mo | $4,842/mo | $20,270/mo | $243K/yr |
| 15,000 developers, light usage | 22.5M | $251,100/mo | $35,508/mo | $215,600/mo | $2.59M/yr |
| 15,000 developers, heavy usage | 67.5M | $753,300/mo | $106,524/mo | $646,800/mo | $7.77M/yr |
| 15,000 developers, realistic mix | 35.8M | $399,200/mo | $56,490/mo | $342,800/mo | $4.11M/yr |
Realistic mix: a standard power-law usage split (10% power users, 60% regular, 30% light) rather than a flat rate — the most defensible single number if you need one to plan against.
Example 1 — routing a routine request to Qwen
The prompt: "Write pytest unit tests for this function, covering the empty-list case and a normal case:
def running_total(xs): ..." The router's complexity classifier scored this well below its escalation threshold — a routine, well-specified task — and sent it to Qwen, the self-hosted model. No frontier call, no escalation. Here's what it wrote:1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
import pytest
def running_total(xs):
total = 0
result = []
for x in xs:
total += x
result.append(total)
return result
def test_running_total_empty_list():
assert running_total([]) == []
def test_running_total_normal_case():
assert running_total([1, 2, 3, 4]) == [1, 3, 6, 10]Both cases the prompt asked for, correct, done. No escalation needed — this is the 74% of traffic this architecture is built for.
Example 2 — routing a genuinely hard request straight to Claude
The prompt: "Design a distributed rate limiter that works consistently across 50 stateless API servers behind a load balancer, with sub-millisecond overhead and no single point of failure. Compare at least two approaches." The same classifier scored this well above the threshold — open-ended systems design, no single correct answer — and routed it to Claude directly, before Qwen ever saw it. That's the routing decision working as intended: cheap model for well-specified work, frontier model for judgment calls, decided automatically per request.
What the quality gap actually looks like
To measure quality, not just routing, we also send the same prompts to both models side by side, off the hot path, and compare. Here's Claude's answer to Example 1's exact prompt:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
class TestRunningTotal:
def test_empty_list(self):
assert running_total([]) == []
def test_normal_case(self):
assert running_total([1, 2, 3, 4]) == [1, 3, 6, 10]
def test_single_element(self):
assert running_total([5]) == [5]
def test_negative_numbers(self):
assert running_total([-1, -2, -3]) == [-1, -3, -6]
def test_mixed_sign_numbers(self):
assert running_total([3, -1, 4, -2]) == [3, 2, 6, 4]
def test_floats(self):
assert running_total([1.5, 2.5, 3.0]) == pytest.approx([1.5, 4.0, 7.0])
def test_does_not_mutate_input(self):
xs = [1, 2, 3]
running_total(xs)
assert xs == [1, 2, 3]
def test_returns_new_list(self):
xs = [1, 2, 3]
assert running_total(xs) is not xsBoth are correct. Qwen covered the two cases the prompt explicitly asked for and stopped. Claude added six more edge cases nobody asked for — negative numbers, floats, immutability — that a thorough reviewer would still want. That's the pattern across our full test set, not just this one prompt: not wrong, just less thorough.
How we scored it: for each test prompt, we took the answer actually served to a real caller, generated an independent fresh answer from Claude to the same prompt, and asked a judge model for one word —
EQUIVALENT, WORSE, or BETTER — on correctness and completeness only, plus a one-sentence reason. Across the test set, the self-hosted model scored 37.5% quality parity (correctness and completeness equal or better than Claude) — on a small sample, so treat it as a real signal rather than a precise metric. Every non-parity case was the same story as above: correct, less thorough, not wrong.Worth validating against your own workload before scaling this to 100% of traffic — and worth noting that closing a completeness gap like this is usually a prompt-tuning problem ("include edge cases" is a one-line instruction), not a model-capability one.
Getting started
You don't need a research team to stand this up. At a high level:
- Deploy vLLM Semantic Router as an Envoy
ext_procsidecar on Amazon EKS (or any Kubernetes cluster) — it ships with sensible defaults for domain/complexity classification out of the box. - Host your open model on Amazon SageMaker AI as a real-time inference endpoint. Any open model that fits your latency/cost target works here; ours was a quantized 27B model on a single GPU instance.
- Create an Amazon Bedrock AgentCore Gateway with two passthrough targets — one pointing at your SageMaker AI endpoint, one at Amazon Bedrock — and one IAM role covering both.
- Point your application at the Gateway's single OpenAI-compatible endpoint. Routing, auth, and escalation all happen behind that one URL; nothing in your calling code needs to know which model actually answered.
The one piece of custom code most setups will need is a small translation sidecar between the router and each backend's native request shape — in our case, a lightweight FastAPI service handling SigV4 signing and protocol translation. Everything else is AWS-managed configuration, not infrastructure you build and operate yourself.
Try it yourself
If your LLM costs are scaling with traffic instead of with task difficulty, this pattern is worth a pilot before it's worth a rewrite: stand up Amazon Bedrock AgentCore Gateway in front of an existing Amazon SageMaker AI endpoint and Amazon Bedrock, put vLLM Semantic Router in front of both, and measure your own offload ratio against your own traffic — the architecture above is a starting point, not a fixed prescription.
- vLLM Semantic Router — the open-source routing layer
- Amazon Bedrock AgentCore Gateway documentation
- Amazon SageMaker AI real-time inference endpoints
Questions about adapting this to your own model or traffic profile? Reach out — happy to walk through the architecture in more depth.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article