Decision Models Explained: where they came from, how to read the leaderboards, and how to choose one on AWS
Part 0 of "Decision models on AWS". A field guide to the small models that answer your agent's bounded questions, and the old ideas they're built from.
Series: Decision models on AWS (3 articles)
- 2Decision Models Explained: where they came from, how to read the leaderboards, and how to choose one on AWS This article
Every request an agent handles hides a handful of small decisions. Which team should get this ticket? Is this message in scope? Should this tool call run, or wait for a person? Does this answer meet the rubric?
None of these needs a paragraph of text. Each needs one option from a list you already know, and ideally a sense of how sure the model is. For the last few years we've mostly asked a large language model (LLM) to make these calls, then parsed its output and hoped it picked one of our options.
In September a new kind of product made that habit look expensive. TypeSafe launched Jev, and within two weeks a wave of open-source alternatives followed, including Strands Decider 2B from Strands Labs at AWS. In Part 1 I measure Decider on five AWS hosts and against Claude Haiku. This part steps back: what these models are, which older ideas they build on, how to read the leaderboards everyone is quoting, and how to choose one for an agent on AWS.
There are already good explainers of Jev's API, so I won't repeat them; I link to them below. This is the guide I wanted when customers started asking me "should we use one of these?"
What's a decision model?
A decision model takes some state (a ticket, a conversation, a planned tool call) and one or more bounded questions, and returns an answer from a fixed set with a probability attached. There are three common question shapes:
- Yes/no: "Is this request in scope?" → a probability between 0 and 1.
- Choice: "Which team should handle it?" → one of the options you listed, with a probability for each.
- Score: "How well does this answer meet the rubric?" → a level on a scale you define.
It can't write prose or hold a conversation. In exchange, it always answers in your format, it's small and fast enough to call on every step, and the probability is meant to be calibrated: when it says 0.8, it should be right about 80% of the time. That last property is what lets you build "decide when sure, escalate when not" into your code.
For the API details, TypeSafe's launch post and documentation are the primary sources. TypeSafe calls these "System One" models, a name that echoes the fast, intuitive mode of thinking in Daniel Kahneman's Thinking, Fast and Slow.
Old parts, new contract
The first reaction from many engineers was "isn't this just a classifier?" It's a fair question. Almost every part of a decision model existed before 2026:
| Year | Idea | What it contributed |
|---|---|---|
| 2015 | Pointer Networks | Answer by pointing at one of the inputs instead of generating text: the idea behind today's "pointer heads" |
| 2018 | BERT | Fine-tuned text classifiers became cheap and accurate, but needed labelled data for every task |
| 2019 | Zero-shot classification as entailment | Classify against any label set with no training, by asking "does this text imply this label?" |
| 2022 | Reward models (InstructGPT) | A model whose only job is to score an output |
| 2022 | SetFit | Usable classifiers from a handful of labelled examples |
| 2023 | LLM-as-a-judge | Ask a strong LLM to score or compare. Flexible, but slow and costly per decision |
| 2023 | Llama Guard | An LLM fine-tuned to answer one question: is this safe, and if not, which policy does it break? |
| 2024 | RouteLLM | Learned routers that decide which model should handle a request |
| 2024 | ModernBERT | Faster encoders with longer context, now the base of several decision models |
So what did Jev change? Not the parts, but the contract:
- One general model, any bounded question. You don't train a classifier per task. You describe the options in plain language at request time.
- Calibration as a product feature. The probability is the point, not a by-product, so you can set thresholds in code.
- Classifier-like price and latency. TypeSafe lists $0.042 per million input tokens with output tokens free, and 70–500 ms end to end, with up to 64k tokens per request. Treat these as vendor figures; Jev is in early access.
That packaging is why the idea spread so fast. Within days there were open reproductions, several of them speaking the same
/v1/systemone request format, and on 1 October Strands Labs released Strands Decider 2B under Apache-2.0.Two ways to sort them
"Decision model" now covers very different things. I find two questions sort them quickly: what job is it doing, and how is it built.
By job
| Job | Typical question | Where it sits in an agent |
|---|---|---|
| Route | "Which team, model or sub-agent should take this?" | At the front door |
| Gate | "Should this tool call run without a human?" | Before an action |
| Guard | "Does this message break our content policy?" | On input and output |
| Judge | "How well does this answer meet the rubric?" | In evaluations and quality checks |
The job matters because it sets the cost of a wrong answer. A misrouted ticket costs a hand-off. A wrongly approved refund costs money. As argued in Jev Isn't Another LLM , a probability should never replace IAM permissions or business rules: the model can recommend, but deterministic controls still decide what's allowed.
By build
| Build | How it answers | Examples (as of 6 Oct 2026) | Good at | Watch out for |
|---|---|---|---|---|
| Hosted decision API | You send state and questions to a vendor's endpoint | Jev | No hosting; strong general accuracy | Your data leaves your account (TypeSafe says its service is currently based on the West Coast); vendor terms; early access |
| Decoder + pointer head | A small LLM reads the state; a small head scores each option | Strands Decider 2B, Kev | Long state, open weights, runs in your account | Needs a GPU for real-time use (Part 1: about 0.13–0.5 s on a GPU, 4–9 s on CPU) |
| Encoder + decision head | A BERT-style encoder reads state and options together | Laya | Fast, CPU-friendly | Shorter context; less general reasoning |
| Logit readout, no training | Ask a general open model and read the probabilities of each option | SemIf | Works with any open model you already host | Calibration depends on the base model |
| Prompted LLM | Ask an LLM with a forced tool call or structured output | Claude Haiku or Amazon Nova on Bedrock | Best zero-shot accuracy in my test | Highest cost per decision; you get an answer, not a calibrated probability |
Is a decoder better than an encoder? This article tested exactly that: Strands Decider: Why Not an Encoder? . The encoders won clearly on latency (a median of about 35 ms against v19's 65 ms), but the accuracy picture was mixed: one variant did better on individual tasks and worse on the overall benchmark, and the one that edged ahead used about four times the parameters. The conclusion: not worth moving into mainline Decider yet. It's the best primary source on the trade-off.
A short list of open models with a stated licence:
| Model | Licence | Base model | Size | Type |
|---|---|---|---|---|
| Strands Decider 2B (v19 at launch, v21 now the reference) | Apache-2.0 | Qwen3.5-2B-Base | ~2B | Decoder + pointer head |
| Kev | Apache-2.0 | Qwen2.5, Qwen3 and Qwen3.5 bases | 0.5B–27B | Decoder + pointer head (the 27B is a full fine-tune) |
| Laya | Apache-2.0 | ModernBERT-large (multilingual: mmBERT-base) | 421M (322M) | Encoder + decision head |
| SemIf | MIT | An open LLM you choose (Qwen3.5-4B in its quickstart) | Depends | Logit readout, no training |
New models appear weekly. Check the model card for licence and base model before you build on one; several popular ones don't state a licence at all.
How to read the leaderboards
Decision-model benchmarks are young, and that shows. Four things tripped me up:
- Two unrelated benchmarks share a name. JevBench on GitHub , with results on Benchmark Heaven, scores typed decisions on intelligence, calibration, speed and cost; it's the one Strands, Laya and most model cards cite. jevbench.dev is a different leaderboard that ranks models by wins in real games. Check which one a claim refers to.
- Versions and scales drift. Decider alone has been quoted as 0.72, 0.76 and 29.5, depending on the checkpoint, the benchmark version and the scoring rule. Compare numbers only within one version of one benchmark.
- Names collide.
Mapika/decider-2bon Hugging Face is an independent open reproduction of Jev, built on the same Qwen base. It isn't Strands Decider. - Accuracy isn't the number that matters most. If you plan to escalate uncertain answers, what you need is coverage at a fixed error rate: what share of requests can the model answer alone while staying, say, 97% accurate? In Part 1, Decider answered 32% of requests alone at 97% accuracy, and 45% at 89%.
And the most useful benchmark is still a few hundred of your own labelled examples. Public leaderboards rank general ability; your routing set measures your problem.
Choosing one on AWS
Here's the order of questions I now go through with customers:
- How often do you make this decision? At low volume, ask an LLM on Amazon Bedrock with a forced tool call and stop there. It's the most accurate option with no tuning, and there's nothing to host.
- Can the state leave your account or Region? If not, a hosted API is out. Run open weights in your own account: a GPU on Amazon SageMaker AI or Amazon Bedrock AgentCore Runtime Instances for live agents, or AgentCore microVMs or AWS Lambda for background work. Part 1 has the measurements and the traps.
- What's a wrong answer worth? For gates and guards, choose a threshold from coverage at your error budget, escalate the rest to an LLM or a person, and keep IAM and business rules in charge.
- Can you use the licence and the base model? Check the model card. Some regulated customers ask where a base model comes from; have that answer ready.
- Can you tune it? Zero-shot, a 2B decision model loses to a strong LLM. A few thousand of your own labels is where I expect that to change. That's a test for this series, not a promise.
Where the series goes from here
Part 1 put numbers on the trade-off for one model and one task:
- 72% vs 93%: zero-shot routing accuracy, Decider 2B against Claude Haiku 4.5.
- 90%: accuracy when Decider answers only when it's confident (45% of requests) and Haiku takes the rest.
- About 36% lower cost than Haiku alone at steady volume, and the confident half never leaves your account.
Next, Part 2 looks for the cases where a decision model clearly wins: several questions in one pass, and a break-even calculator for decision models against LLM tokens.
Series: Decision models on AWS (3 articles)
- 2Decision Models Explained: where they came from, how to read the leaderboards, and how to choose one on AWS This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article