Testing Decision models - Deep Dive into Decision Models
Decision models promise fast, cheap, reliable choices, so I put nine of them to the test. Using 1,000 open cases, I compared them on two tasks: routing a request to the right tool, and matching a record to the same person in a list. Every model ran on Amazon Bedrock or a SageMaker real-time endpoint, and I tracked accuracy, latency and cost per decision.
Series: Testing Decision Models (2 articles)
- 2Testing Decision models - Deep Dive into Decision Models This article
In Part 1 , I compared three ways to handle a classification step by prompting a small language model, using TypeSafe's Jev decision model, and training a traditional classifier on labeled data. That test used spam detection with two possible answers. It answered a useful question, but it left me with a bigger one. Was Jev strong because it fit that one task, or was there something useful in this new category of decision models?
The timing made the question hard to ignore. TypeSafe introduced Jev as the first "System One" model, one that reads some state and returns typed decisions and probabilities instead of generating a response. Soon after, open models such as Laya, Strands, and CLM appeared or came to light with similar interfaces. The releases came with claims about lower latency, lower cost, calibrated confidence, and a better fit for software workflows than a normal language model.
That was enough activity to justify a closer test, but not enough evidence to make an architecture decision. New benchmarks appear every day across different categories. Their results are often measured on different data, hardware, and definitions of success. I wanted to put the available approaches behind one input and output contract, score them on the same cases, and see what differences were comparable.
Recap: what a decision model is
A decision model takes a state, a question, and a set of possible answers. It scores those answers and returns one of them, often with a probability distribution or confidence value.
A support system might receive this state:
The customer says their card was charged twice and asks for a refund
The customer says their card was charged twice and asks for a refund
Then it might ask:
Which team should handle this?
Which team should handle this?
- billing: invoices, charges, and refunds
- technical: bugs, outages, and errors
- sales: pricing and new contracts
A decision model will return the selected option directly. Models such as Strands, and Laya will read the supplied options in one model pass instead of generating an answer token by token. CLM will take a related approach by turning the state and candidate actions into representations, then compare them.
Model Roster
The roster covered several ways to place a decision inside an application:
| Model | Foundation used in this test | How it makes the decision |
|---|---|---|
| Jev | Internal architecture is not published | TypeSafe exposes a typed decision API trained to return calibrated probabilities rather than generated text |
| Laya | ModernBERT-large | A custom decision head scores option markers in one pass |
| Laya typed decisions | ModernBERT-large | A separate checkpoint trained for typed decision tasks |
| ModernBERT NLI | ModernBERT-base | An inference classifier scores how strongly the input supports each candidate statement |
| Strands Decider 2B | Qwen3.5-2B | A pointer head selects one supplied option instead of generating its text |
| Decider 4B | Qwen 3.5-4B-Base | It reads option scores at a designated answer position |
| Plumb 4B | Qwen3.5-4B through the JevK5 lineage | It returns probabilities over the supplied options in one pass |
| CLM | Frozen Qwen3-8B backbone used for embeddings | Learned state and action projection heads map the request and candidates into a shared space for comparison |
| Qwen3-8B base | Qwen3-8B | A normal generative baseline prompted to return one candidate ID |
| GPT-6 Luna | Hosted generative model | A constrained prompt asks for one candidate ID through Amazon Bedrock |
| GLM-5.3-Flash | Hosted generative model | A constrained prompt asks for one candidate ID through Z.ai |
As for the reasons I included these models, Laya was the clearest open ModernBERT-based alternative, while ModernBERT NLI gave me a simpler encoder baseline. Strands represented the Qwen-based open decision-model branch at different sizes and with different readout designs that I also wanted to compare against Jev.
CLM gave me a contrastive design that could be adapted with a small trainable head. Untuned Qwen3-8B was the control for asking a normal open model to make the same choice. Luna and GLM was picked so I could test whether a general hosted model could match the accuracy while accepting the cost and latency of generation.
I also wanted more pressure than the binary spam test in Part 1. What happens with 2 choices, 8 choices, or 20? Does an accurate model stay fast with more choices? Does a cheap API still look cheap next to a model I host myself? What does fine-tuning buy when the base model has a clear weakness?
Two foundation families: ModernBERT and Qwen
As you can see in the model roster, there is a common foundation family across these types of models. I think it's important to understand the differences between these models.
ModernBERT is an encoder-only model. It reads the full input in both directions and builds a representation that can support classification, retrieval, and matching. I think of it as a reader and grader. That makes it a sensible foundation for a fast decision model, but it is one branch of this category rather than the category itself.
Qwen is a family of open-weight language models developed by Alibaba Cloud's Qwen team. Qwen3-8B is a decoder-only generative model. Its native task is predicting the next token, which makes it a reader and writer. Several models in this benchmark adapt Qwen-family backbones to select from fixed candidates instead. Their output is constrained even though their foundations are generative.
Laya uses ModernBERT-large with a custom decision head. It scores each answer at an option marker, then turns those scores into a probability distribution. ModernBERT NLI is a separate ModernBERT-base checkpoint which is Laya's foundation. It was trained for natural-language inference, where it scores whether one statement supports another. I used it as a baseline by turning every candidate into a hypothesis and selecting the one with the highest support score.
That distinction mattered to the benchmark design.
Comparing Laya with ModernBERT NLI helped separate the value of a decision-specific head from the ModernBERT models. Comparing CLM with untuned Qwen3-8B showed what a task-specific projection head added to the same foundation family. Strands covered different attempts to turn a Qwen-style model into a one-pass decider. Jev represented the hosted, purpose-built API that started the discussion. Luna and GLM represented the option of using a general model already available through an API.
How I deployed the models
I used three deployment paths because models don't all come in the same form.
Hosted APIs
Jev ran through TypeSafe's
/v1/systemone API. I sent a state and a typed choice question with candidate criteria, and the API returned a candidate ID directly. Jev was priced by input token, with output free at the rate used for this comparison.GPT-6 Luna ran through the Amazon Bedrock Converse API using the
us.openai.gpt-6-luna cross-Region inference profile. Bedrock gave me an AWS-authenticated API, usage reporting, and access to a managed model without deploying GPU infrastructure. I used explicit connection and read timeouts so a slow response wouldn't be mistaken for a failed request and retried unnecessarily.GLM-5.3-Flash ran through Z.ai's chat-completions API. For Luna and GLM, I formatted the task as a constrained prompt and required the exact candidate ID as the response. The harness recorded an empty response, unknown candidate, multiple candidate IDs, or transport error separately.
SageMaker real-time endpoints
I used Amazon SageMaker real-time inference for CLM, Strands, Plumb, Decider, and the untuned Qwen baseline. I wanted a warm process for each model so I could measure application-visible latency without loading weights for every case. A batch job would have been cheaper for a one-time accuracy sweep, but it wouldn't have represented a synchronous routing call inside an application.
Local models
I ran ModernBERT NLI, Laya, and Laya typed decisions locally. This let me include the smaller models without maintaining three more endpoints. Their chart cost is $0 billed for this experiment, so strong note that this isn't a correct measure for hosting in prod.
ModernBERT NLI scored each candidate as an entailment hypothesis and selected the highest entailment score. Laya used its typed-decision head. I sized Laya's input budget to the number of candidates so longer lists weren't impacted by a fixed default.
What I benchmarked and why
I tested 11 model configurations on 1,000 cases from DecisionBench , then fine-tuned CLM as a twelfth result. I measured exact-match accuracy, median and 90th-percentile latency, completion failures, and estimated inference cost for the same 1,000 decisions.
I figured it would be a good start to test tool routing because that is where I see a lot of classification use-cases shining recently. JevBench is the benchmark most decision-model authors report against, which is the part of the reason I didn't use it.
There are some public results from JevBench that gets used as some training recipes for models, and I figured I wanted to start off quirky and test for other bencharks I found.
I chose two categories from this bench that exercise different structures:
software_agents, specifically tool and skill routing drawn from the ToolACE dataset . The input describes a request, and the model chooses the best tool from a list. This matches a practical use case for a decision model inside a software workflow.public_sector, specifically entity alignment drawn from Dutch law-enforcement entity-resolution. The input contains one record, and the model chooses the matching person or entity from several candidate records. This tests record comparison rather than intent-to-tool matching.
I used 500 cases from each domain. Candidate count can make the same task much harder, so I divided each domain into four buckets which were 2 candidates, 3 to 4, 5 to 8, and 9 to 20. Each domain and candidate-count cell contributed 125 cases.
The expected answer was fixed by the dataset. Every model received the same state, instruction, candidate IDs, complete candidate text, and candidate order.
The goal was to compare architectural choices using one contract with read a state, choose one candidate, and return an answer my code can use.
What I found
Six models finished between 98.8% and 99.9% accuracy. Their 95% confidence intervals, so this test supports broad tiers rather than a precise ranking within the top group.
| Model | How I ran it | Accuracy | Median latency | Cost per 1,000 decisions |
|---|---|---|---|---|
| GLM-5.3-Flash | Z.ai API | 99.9% | 3,107 ms | $0.173 |
| Jev | TypeSafe API | 99.9% | 236 ms | $0.042 |
| GPT-6 Luna | Amazon Bedrock | 99.9% | 1,041 ms | $0.084 |
| Qwen3-8B base | SageMaker real-time endpoint | 98.8% | 836 ms | $0.318 at 100% utilization |
| Strands 2B | SageMaker real-time endpoint | 95.7% | 154 ms | $0.067 at 100% utilization |
| Plumb 4B | SageMaker real-time endpoint | 99.9% | 190 ms | $0.083 at 100% utilization |
| Decider 4B | SageMaker real-time endpoint | 99.6% | 187 ms | $0.084 at 100% utilization |
| ModernBERT NLI | Local CPU | 93.6% | 109 ms | $0 billed locally |
| Laya typed decisions | Local CPU | 78.6% | 131 ms | $0 billed locally |
| Laya | Local CPU | 64.1% | 128 ms | $0 billed locally |
| CLM | SageMaker real-time endpoint | 54.8% | 169 ms | $0.086 at 100% utilization |
| CLM, fine-tuned | Same CLM serving path | 70.2% | Not measured separately | Same inference cost as CLM |
Every model returned a valid answer for all 1,000 cases. I counted any missing, malformed, ambiguous, or failed response as incorrect instead of removing it from the denominator.
Accuracy and latency separate the top tier
The latency view makes the top group easier to compare.
The fastest observed result came from local ModernBERT NLI at 109 milliseconds, followed by local Laya and Laya typed decisions. Their accuracy was lower. Strands reached 154 milliseconds on SageMaker and scored 95.7%, which may still be a good trade when the application values speed and control over the serving path.
Important note here is that these latency measurements came from different hosting paths and worker counts, so I used them as application-level observations.
Accuracy and cost show a different trade-off
Jev was the cheapest billed model in the top accuracy tier at $0.042 per 1,000 decisions. Luna was near $0.083. GLM and Qwen base cost more without improving measured accuracy. Strands was fast and inexpensive at full endpoint utilization, but its 95.7% accuracy put it in a different tier on this test.
The three local CPU models appear at $0 because nothing was billed for those runs. That does not make production hosting free. The SageMaker figures also assume the instance stays busy. Sparse traffic changes the calculation because an endpoint continues billing while it waits for work, while a token-priced API charges only when called.
Domain and candidate count still mattered
The per-domain results explain some of the separation:
| Model | Software routing | Entity alignment |
|---|---|---|
| Jev | 99.8% | 100.0% |
| Luna | 99.8% | 100.0% |
| GLM-5.3-Flash | 99.8% | 100.0% |
| Qwen3-8B base | 99.0% | 98.6% |
| Strands 2B | 99.2% | 92.2% |
| Plumb 4B | 99.8% | 100.0% |
| Decider 4B | 99.6% | 99.6% |
| ModernBERT NLI | 93.2% | 94.0% |
| Laya typed decisions | 64.0% | 93.2% |
| Laya | 40.2% | 88.0% |
| CLM | 81.4% | 28.2% |
Laya and CLM show why one overall number is not enough. Laya handled entity alignment much better than software routing. CLM did the opposite. Choosing either model from its aggregate score would hide the part that matters for a specific workload.
Candidate count exposed another difference. Base CLM fell from 92.0% to 70.4% on software routing as the choices increased from 2 to 9 through 20.
On entity alignment, it fell from 54.4% to 9.6%. Laya's token budget also changed its result because longer candidate lists competed for a fixed input budget. I reported one row per Laya checkpoint using a budget sized to the number of options and kept the checkpoint-default runs outside the main table.
The top models saturated on these two families, which limits what the benchmark can say about differences inside that tier. The entity-alignment cases were also solvable with direct attribute matching when the fields were structured. For a production workflow with the same shape, I would compare any model against that simpler deterministic baseline before deploying one.
Fine-tuning improved CLM on the measured workload
I wanted one clear example of fine-tuning rather than a large collection of tuning experiments. CLM was the best candidate because the base model had room to improve and its architecture supports small trainable projection heads over frozen Qwen3-8B embeddings.
2,800 training questions were built and split evenly between the same software-routing and entity-alignment families. No training row shared an ID or state text with an evaluation case. Of those questions, 739 were original rows already in the 2-to-20 candidate range. For the other 2,061, I used larger DecisionBench rows and kept the correct option plus randomly selected distractors to produce the same candidate-count range as the evaluation.
Qwen3-8B frozen was kept frozen. CLM used it to produce embeddings for the request and each candidate. A trainable state head and action head mapped those embeddings into a shared space. The training objective moved each request closer to its correct candidate and farther from the distractors.
In this testing, I used four heads using InfoNCE and softmax cross-entropy, each with two seeds. The winning run used InfoNCE and reached 73.8% validation accuracy at epoch 20. I chose it using validation accuracy only.
The final result moved overall accuracy from 54.8% to 70.2%. It fixed 209 cases and broke 55.
| CLM result | Before fine-tuning | After fine-tuning |
|---|---|---|
| Overall | 54.8% | 70.2% |
| Software routing | 81.4% | 97.4% |
| Entity alignment | 28.2% | 43.0% |
| Software routing, 9 to 20 choices | 70.4% | 96.0% |
| Entity alignment, 9 to 20 choices | 9.6% | 22.4% |
The training data came from the same two task families as the evaluation, using different records. Software routing became nearly solved. Entity alignment improved by 14.8 points but remained far behind the top group.
The training ran as a SageMaker Training job on one
ml.g5.2xlarge instance. It cost about $0.86 at the on-demand rate used in the test. Preparing a valid training set and protecting the evaluation took more care than the training.What model to choose?
For the workload I measured, I would choose Jev when accuracy, billed cost, and low latency matter together. It reached 99.9%, cost about $0.042 per 1,000 decisions, and answered in 236 milliseconds at the median.
Maybe choose a SageMaker endpoint when control over the model and serving path matters more than avoiding fixed cost. That choice also means managing endpoint capacity, scaling, monitoring, and idle cost.
Consider ModernBERT NLI when 93.6% is sufficient and if compacting model that can run without a paid external inference call is necessary. This experiment placed it in a lower accuracy tier, but that may still be the right operational choice.
Use a CLM fine-tuning when there are representative labeled examples for a stable decision family and value customization. Its software-routing gain shows what a small task-specific head can learn. Its entity-alignment result shows that the same training method does not make the model equally strong on every task.
Luna or GLM can be used when the decision step may grow into work that needs broader generation or reasoning and keeping one model interface is important. On this benchmark, that flexibility added latency without improving accuracy.
The lesson I am taking forward
Models for structured decisions will keep arriving. Small models will get faster, open checkpoints will be adapted for new workflows, and a result from this month will change in the next. I hear at the time of my testing that Quyet-1.0 has become a top contender and that OpenAI announced its Decisions API at DevDay 2026. It is in limited preview and uses Luna to classify inputs, route requests, or choose an action from predefined answers.
The stable part and part that seems to matter will always be benchmarking. Use cases that look like the decisions your application makes, candidate counts that match production, the complete candidate content, a fixed answer key, and failure handling that does not hide errors. Then measure the dimensions that can change the architecture including accuracy by task family, latency, token or instance cost, and utilization.
There is no single right model because there is no single definition of right. The answer changes with the workload, traffic pattern, deployment boundary, and cost of a mistake. For this benchmark, Jev offered the strongest combined accuracy, billed cost, and latency. For another application, the same measurements may point somewhere else...until the next week we have a new model or API to play with as a competitor.
Series: Testing Decision Models (2 articles)
- 2Testing Decision models - Deep Dive into Decision Models This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article