When does a decision model pay for itself?
Strands Decider 2B and Claude Haiku 4.5 compared on four questions per support message, measured on Amazon SageMaker AI and priced from 100 to 25,000 requests an hour. It includes the break-even point against one LLM call, an inline cost calculator, and why self-hosting rarely beats hosted decision APIs on price.
Series: Decision models on AWS (3 articles)
- 3When does a decision model pay for itself? This article
In Part 1 I ran Strands Decider 2B on five AWS hosts. On one routing question it lost to Claude Haiku 4.5, 72% against 93%. Letting Decider answer when it was confident and handing the rest to Haiku got 90%, at about 36% lower cost. Part 0 covered where decision models came from and how to choose one.
The question I keep getting since is simpler: is it actually cheaper than an LLM, and at what volume?
Real agents rarely ask one question. A support message needs routing, a scope check, a personal-data check and a "does a person need to see this?" check, often as four separate model calls. Decision models can answer several questions in one pass, so that's where I expected Decider to win. I wrote down three pass/fail bars before running anything, then measured.
Two things changed between Part 1 and this test:
- Hosted decision APIs now have public prices. OpenAI's Decisions API went into public beta on 6 October, and TypeSafe lists Jev's price per input token.
- Strands published a new checkpoint,
hobson-v21, with the same accuracy as v19 and better calibration. I used v21, with v19 as a check.
Batching gets half an answer here: several questions in one request is measured below; batching separate requests on the GPU isn't possible yet, for reasons in the gotchas.
The results first
| Bar, written before the test | Result |
|---|---|
| A. Four questions cost Decider at most 1.3× the latency of one | Fail: 196 ms against 71 ms, 2.8× |
| B. Decider first, Haiku when unsure: within 3 points of Haiku's accuracy and at least 40% cheaper than four separate Haiku calls, at 1 request a second | Pass: 70% against 73% all-four-right, 58% cheaper ($1.69 against $4.06 per 1,000 requests) |
| C. The same, against one Haiku call answering all four questions | Fail: only 4% cheaper ($1.69 against $1.77) |
So, when does a decision model pay for itself?
- Against four separate LLM calls: almost always, above about 500 requests an hour on one GPU.
- Against one well-built LLM call: only at volume. Break-even is about 3,000 requests an hour on a SageMaker AI ml.g6.xlarge, or about 1,250 if only the routing question goes to the LLM.
- Against a hosted decision API: not on price, at any volume one GPU can serve. Self-hosting buys you residency and control, not a lower bill.
The surprise was that Decider beat Haiku on one question outright, and that how you split the questions matters as much as which model answers them.
The test
200 synthetic support messages for a fictional invoicing SaaS: the 100 from Part 1 plus 100 new ones, 70 of them deliberately ambiguous (90 hard in total). Each message has four questions:
| Question | Type | Positive labels |
|---|---|---|
| Which team should handle it (billing, technical, sales, account, unknown)? | choice | — |
| Is it about our product? | yes/no | 171 of 200 |
| Does it contain personal data? | yes/no | 17 |
| Should it skip the bot and go straight to a person? | yes/no | 37 |
To keep the answer key independent of the Claude model I compare against, the final labels come from a different model family: OpenAI's GPT-5.6 labelled all 200 items blind against written rules. A Claude-drafted key agreed on 97–100% per question. For the 10 items where they differed, I used GPT-5.6's label. All the text is synthetic; names, emails and bank details are fictional.
The arms, on the same 200 items:
- Decider v21, all four questions in one request (and v19 for comparison)
- Decider v21, one question per request
- Claude Haiku 4.5, one forced tool call answering all four. This is the fair baseline.
- Claude Haiku 4.5, four separate calls. This is how guardrail, router and gate stacks are often built.
- Decider first, Haiku below a confidence threshold, scored from the runs above
Decider ran on one SageMaker AI real-time endpoint on ml.g6.xlarge (NVIDIA L4) in eu-west-1. Haiku used the EU cross-Region inference profile at temperature 0. Both got the same wording for every question and team.
Four questions for the price of one?
A four-question request to Decider looks like this:
1
2
3
4
5
6
7
8
9
10
11
body = {
"state": f"A customer sent this message to AnyCompany Invoicing support:\n\n{text}",
"questions": {
"route": {"type": "choice", "instructions": "Which team should handle this message?",
"criteria": TEAMS}, # five teams, one line each
"in_scope": {"type": "noul", "instructions": "Is this message about the AnyCompany Invoicing product?"},
"personal_data": {"type": "noul", "instructions": PERSONAL_DATA_RULES},
"needs_human": {"type": "noul", "instructions": NEEDS_HUMAN_RULES},
},
}
answers = decider(body)["answers"] # one choice with probabilities, three P(true) valuesDecider's docs say asking many questions is "nearly free", because the state is encoded once and shared across questions. On my test, it wasn't:
| Model time, p50 | 1 question | 4 in one request | 4 separate requests |
|---|---|---|---|
| Decider v21 on ml.g6.xlarge | 71 ms | 196 ms | 273 ms |
| Claude Haiku 4.5 (as reported by Bedrock) | 519 ms | 768 ms | 2,149 ms |
Why not nearly free? Our messages are short, and our questions aren't. Each yes/no question carries its own instructions plus about 65 tokens of template, so four questions took 572 input tokens against 236 for routing alone. With one-line wording for every question, four still took 2× as long as one (132 ms against 66 ms). The "nearly free" result holds when the state is long and the questions are short: a long document with a few quick checks, not a one-line message with detailed rules.
One request is still better than four. It's 28% faster, and it raised throughput on the L4 from 3.6 to 5.0 requests a second.
Accuracy per question
| Route | In scope | Personal data | Needs a human | All four right | |
|---|---|---|---|---|---|
| Claude Haiku 4.5, one call | 94% | 99% | 99% | 78% | 73% |
| Claude Haiku 4.5, four calls | 92% | 95% | 100% | 71% | 64% |
| Decider v21, one request | 69% | 93% | 94% | 84% | 48% |
| Decider v19, one request | 69% | 94% | 94% | 89% | 52% |
| Decider first, Haiku below 0.6 | 94% | 94% | 95% | 84% | 70% |
Three things stand out:
- Routing is still Decider's weak spot. Like in Part 1, it under-predicts "sales" (12 predicted against 35 labelled) and over-predicts "unknown". It's only slightly worse on the hard items (67%) than the easy ones (71%), which suggests the option descriptions are the problem, not ambiguity.
- On "needs a human", Decider scores higher than Haiku, but read the errors. Of the 37 messages that needed a person, Haiku flagged 35 and Decider 25. Haiku's lower score comes from over-flagging: 42 false alarms against Decider's 21. If missing an angry customer costs more than a wasted handoff, Haiku's errors are the better kind. Accuracy alone would have hidden that.
- Splitting questions makes the LLM worse, not just dearer. Asked one question per call, Haiku lost 9 points on all-four-right. With every question in one call, it can see the whole picture: a complaint about a charge is both "billing" and "needs a human".
Decider gave almost the same answers in one request and in four (48% all-four-right either way), and the L4 matched a run on my laptop on 199 of 200 routes. Where it runs changes speed and cost, not answers.
Escalation compounds
In Part 1, at a threshold of 0.6, 55% of requests went to Haiku. With four questions it's 76%, because a request escalates if any question is unsure:
| At 0.6 | Route | In scope | Personal data | Needs a human | Any question |
|---|---|---|---|---|---|
| Share below threshold | 71% | 2% | 2% | 21% | 76% |
The yes/no questions rarely escalate; routing does. So I tried a variant I hadn't planned (treat it as exploratory): Decider answers everything, and only an unsure routing question goes to Haiku, as a route-only call that costs $1.07 per 1,000 instead of $1.77. At 0.6 that's 68.5% all-four-right, about 5 points under Haiku, at 37% lower cost than one Haiku call. It's the best cost-accuracy trade-off I found, but it still misses bar C.
1
2
3
4
5
6
answers = decider(body)["answers"]
route = answers["route"]
if route["confidence"] \< 0.6: # 71% of requests on this set
route = ask_haiku_route_only(text) # one short LLM call
decision = {"route": route["choice"],
**{q: answers[q]["noul"] \>= 0.5 for q in ("in_scope", "personal_data", "needs_human")}}What it costs, and the calculator
Prices are eu-west-1 list prices on 7 October:
- Claude Haiku 4.5: $1 per million input tokens, $5 per million output
- SageMaker AI ml.g6.xlarge: $1.258 an hour
- AgentCore Runtime Instances on g4dn.xlarge: $0.587 an hour plus the 7.8% management fee
Haiku used about 1,320 input and 90 output tokens for one four-question call.
| Per 1,000 four-question requests, at 1 request a second | Cost |
|---|---|
| Claude Haiku, four separate calls | $4.06 |
| Claude Haiku, one call | $1.77 |
| Decider first, Haiku when any question is unsure (76%) | $1.69 |
| Decider first, only an unsure route to Haiku (71%) | $1.11 |
| A fully used ml.g6.xlarge on its own (5 requests a second) | $0.07 |
The whole model fits in two functions. Plug in your own volume, prices and escalation rate:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
import math
def cost_per_1000(requests_per_hour, instance_per_hour, max_rps, escalation, llm_cost_per_request):
"""$ per 1,000 requests for 'decision model first, LLM when unsure' on always-on instances."""
instances = max(1, math.ceil(requests_per_hour / (max_rps * 3600)))
gpu = instances * instance_per_hour / requests_per_hour * 1000
return gpu + escalation * llm_cost_per_request * 1000
def break_even(instance_per_hour, escalation, llm_cost_per_request, escalated_cost=None):
"""Requests per hour above which the cascade beats sending everything to the LLM."""
escalated_cost = llm_cost_per_request if escalated_cost is None else escalated_cost
saving = llm_cost_per_request - escalation * escalated_cost
return math.inf if saving \<= 0 else instance_per_hour / saving
haiku_one_call = (1317 * 1.00 + 90 * 5.00) / 1e6 # $ per request
print(cost_per_1000(3600, 1.258, 5.0, 0.76, haiku_one_call)) # ≈ 1.69
print(break_even(1.258, 0.76, haiku_one_call)) # ≈ 2,970 requests/hour
print(break_even(1.258, 0.71, haiku_one_call, 0.00107)) # route-only ≈ 1,250| Break-even against one Haiku call | Decider alone (48% all-four-right) | Decider first, Haiku when unsure (76%) |
|---|---|---|
| SageMaker AI ml.g6.xlarge | about 710 requests/h | about 2,970 requests/h |
| SageMaker AI ml.g4dn.xlarge | about 465 requests/h | about 1,940 requests/h |
| AgentCore Instances g4dn.xlarge | about 360 requests/h | about 1,490 requests/h |
The T4 rows break even sooner because the instance is cheaper. They also top out sooner: Part 1 measured the T4 about 3.7× slower than the L4, so plan on roughly 1.3 four-question requests a second per T4. That's an estimate, not a measurement.
The escalation rate matters more than the GPU. Every point of escalation you remove saves about 1.8 cents per 1,000 requests at any volume; the GPU's share falls towards zero once it's busy. That's why fine-tuning on your own labels, which should make Decider confident more often, is the next test.
Against hosted decision APIs
At Decider's 570 input tokens per request (their tokenisers differ, so treat this as approximate), list prices are:
| Hosted decision API | List price | Per 1,000 four-question requests |
|---|---|---|
| OpenAI Decisions API (public beta, 6 Oct) | $0.10 per million input tokens | about $0.06 |
| TypeSafe Jev | $0.042 per million input tokens | about $0.02 |
I didn't measure their accuracy on this set; this is price only. At about $0.06 per 1,000, a self-hosted GPU only matches them above roughly 20,000 requests an hour, which is more than one instance can serve. If price is all you're optimising, a self-hosted decision model doesn't pay for itself against these APIs.
The reasons to run it in your own account are the ones from Part 1. For the regulated startups I work with, they're usually the deciding ones:
- prompts and decisions stay in your AWS account and Region
- access is controlled by IAM
- there's no third-party processor to add to your data-protection paperwork
- you can fine-tune on your own labels
Which should you pick?
| If… | Use |
|---|---|
| You make several decisions per request, under about 1,000 requests an hour | One structured Claude Haiku call. Not one call per question |
| You run more than about 1,500–3,000 requests an hour and data must stay in your account | Decider first, Haiku for unsure answers, on a GPU you keep busy |
| Routing is the hard question and the yes/no checks are easy | Decider for the yes/no checks, Haiku for unsure routes only |
| Missing a case is expensive (a person must see it) | Check recall, not accuracy. Keep the LLM on that question, or lower the threshold |
| Price is all that matters and a third-party API is acceptable | A hosted decision API |
Gotchas
- Lock the engine. The released
strands-decider(0.1.0) keeps per-request state on the engine (issue #17 ). Two overlapping requests can get each other's answers, with HTTP 200 and no error. Until a fix ships, serialise calls, and pin the commit you test:```python
import threading
LOCK = threading.Lock()@app.post("/invocations")
def invocations(body: dict):
with LOCK: # one request at a time; overlapping calls can mix up answers
out = ENGINE.evaluate(SystemOneRequest(**body))
return out.model_dump() - Set thresholds per question, not globally. One 0.6 threshold sent 71% of routes to Haiku but only 2% of the scope checks.
- Count the LLM's tokens honestly. A structured call with four field descriptions used 1,320 input tokens, about 50% more than Part 1's two-question call.
- AWS has an official starting point. If you're hosting Decider on SageMaker AI, there's now an aws-samples example . It serialises engine access too.
Cost and clean-up
This part cost under $5:
- about 1.5 hours of ml.g6.xlarge
- about 1,200 Haiku calls
- one AWS CodeBuild image build
I created the endpoint with a small script rather than a stack and deleted it the same day, along with its IAM role.
Series: Decision models on AWS (3 articles)
- 3When does a decision model pay for itself? This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article