System One Models: Strands Decider 2B on the same tests I ran on Jev last week
What Strands Decider is, how it differs from Jev, and what I saw when I ran my fraud, routing and refund tests on it without changing the data or the questions.
Last week I tested Jev , TypeSafe AI's decision model, on fraud triage, model routing and refund decisions. This week the Strands team released Strands Decider 2B , an open source model.
So I ran the same tests again using Strands Decider 2B instead of Jev - the same data, the same questions, the same code. In this article I explain what Strands Decider is and how it differs from Jev. I show the same request answered by both models, then walk through the use cases and what I observed.
What Strands Decider is
Strands Decider is a decision model. Like Jev, it does not write text. You give it some data and a list of questions, and it picks one of the answers you allowed, with probabilities attached. It uses the same three question types: choice (pick one option), noul (yes or no) and score (a point on a scale).
The difference is where it runs and what you get. Strands Decider is a 2 billion parameter model from Strands Labs. The weights are on Hugging Face, and the code, training data and training scripts are on GitHub under the Apache 2.0 licence. I am running an instance of the model locally on my laptop.
Strands also publishes how it works. They took a pretrained LLM, Qwen3.5-2B, and removed the part that writes words, the language-model head. In its place they added a small "pointer head" of about a million parameters. The model reads your data and questions once, and the pointer head scores each option you listed. The answer comes from that single pass, with no text generated.
Because the data is read once, extra questions about the same data cost little. Each question only adds its own tokens.
Where it ran and how I called it
Strands Decider comes with a small server. Starting it takes two commands:
The first start downloads the base model. After that it serves from the local cache
In this test the model was hosted on my laptop, an M4 Mac with 16 GB of memory. Nothing went over the internet. The first request after starting took 10 to 15 seconds while the model warmed up, so I sent a few requests before measuring.
The server takes requests at
POST /v1/systemone, the same path and shape as Jev. You send a JSON body with the data in state, the model name, and your questions:
The transaction and the questions are exactly as they were for Jev
The same request, two decision models
I sent Strands Decider the card-transaction request I had used with Jev. This is the request, unchanged apart from the model name:
Request As Sent to the Strands Decider except the model name
And this is the response:
The same shape as Jev's response. There is no cost field, because nothing is billed.
Side by side:
Both models picked card testing.
- Strands Decider was far less sure. It gave card testing 75% and legitimate 16%. On risk it spread the probability across all five levels, which averages out at 2.30, between moderate and high.
- Jev put 90% on "very high" for the same transaction: tiny charges, many of them in an hour, from a headless browser, on a new account.
- Strands Decider was far less sure. It gave card testing 75% and legitimate 16%. On risk it spread the probability across all five levels, which averages out at 2.30, between moderate and high.
- Jev put 90% on "very high" for the same transaction: tiny charges, many of them in an hour, from a headless browser, on a new account.
Note : The confidence numbers are not computed the same way, so don't compare them directly. Strands documents its formula: the top probability, rescaled so that an even split gives 0 and a certain answer gives 1. Here that is (3 × 0.75 − 1) / 2 = 0.62. For either model, pick the threshold from your own labelled data.
How it differs from Jev
- It runs where you run it. The data never leaves your machine. There is no API key and no price per call. You pay for the hardware instead.
- You can retrain it. The training data and scripts are public. Strands says a full retrain takes about 11 hours on one RTX 3090. I did not try this.
- It is small. Strands reports it gets every easy task on JevBench right, 88% of standard tasks and 51% of hard ones. Easy, rote decisions are its target.
- Speed depends on your hardware. Strands reports a median of 115 ms per question on an RTX 3090 and 153 ms on an M3 Pro MacBook for short tasks.
- My server was for local experiments. It has no authentication. I sent one request at a time.
How I ran it
I used the Python harness I had built for the Jev tests. Each use case is one YAML file with the data, the questions and the thresholds. I added one flag that sends the requests to the local server instead of OpenRouter. The datasets, the input fields, the question wording and the code that turns answers into decisions did not change.
Use case 1: fraud triage
Setup. 100 synthetic card transactions, 20 for each of five classes: legitimate, card testing, account takeover, friendly fraud and merchant collusion. Each transaction is eleven short text fields. Each one was a single request with four questions: the five-way choice, two yes/no questions and a risk score.
What I found.
- Strands Decider got 61 of 100 right. There was no transaction it got right that Jev got wrong.
- Two mix-ups explain most of the misses. It called 12 of the 20 card testing transactions account takeover, and 10 of the 20 merchant collusion transactions friendly fraud.
- It found account takeover and friendly fraud well (19 of 20 each), but it also used those two labels for many transactions that were something else.
- Its confidence still pointed in the right direction. The 9 answers at 0.8 or above were all right. Only 27 of the 100 reached 0.7, and 78% of those were right.
- A request with four questions about one transaction took a median of 1.1 seconds on my laptop.
This was the hardest test: five classes that share signals, judged from eleven short fields.
Why possibly the mix-ups happened?
Card testing transactions that came from an unfamiliar device or a mismatched location were read as account takeover. My account takeover description names "new device, new location". Merchant collusion transactions with a note from the cardholder disputing the charge were read as friendly fraud, which my description defines as a cardholder who disputes.
Why possibly the mix-ups happened?
Card testing transactions that came from an unfamiliar device or a mismatched location were read as account takeover. My account takeover description names "new device, new location". Merchant collusion transactions with a note from the cardholder disputing the charge were read as friendly fraud, which my description defines as a cardholder who disputes.
In both cases the model seemed to match the words in my option descriptions, not the pattern the class is really about. Jev read the same descriptions and got 19 of these 22 transactions right.
A smaller model may need sharper descriptions, such as "many tiny charges in a short time, whatever the device", or fine-tuning on labelled examples. I did not reword the questions for this test
Use case 2: routing prompts between Haiku, Sonnet and Opus
Setup. The same 80 prompts, 20 for each of four tiers. The model read each prompt and picked a tier. The prompt then went to that tier's Claude model: Haiku 4.5 for simple, Sonnet 5 for medium, Opus 5.5 for complex, and Opus 5.5 with extended thinking for reasoning. Opus then judged each answer against a reference answer.
What I found.
- Strands Decider picked the intended tier for 72 of 80 prompts, against 66 for Jev.
- It sent all 20 complex prompts to Opus. Jev had sent 8 of them to Sonnet.
- Its routed answers were judged correct 91–94% of the time across two runs, against 89% for Jev and 97% for always using Opus.
- But it saved only about 5% compared with always using Opus, where Jev saved 25%. Getting complex prompts right means sending them to Opus. Jev's mistakes happened to be the cheap kind.
- Each routing decision took a median of about 300 ms on my laptop.
Choosing the intended tier more often did not save money on this dataset. Half my prompts were complex or reasoning, and those go to Opus either way. With more simple traffic, the savings would come from the simple and medium prompts, as they did with Jev.
Caveats
- All the test data is synthetic, generated by an LLM, and the sets are small (60 to 100 items).
- Each Jev test ran once. I ran the Strands Decider tests twice, and it gave the same answer on every row both times. Only the routed answers from Claude, and their grades, varied.
- I wrote the questions while testing Jev and did not reword them for Strands Decider.
- Jev's latency was measured from my laptop through OpenRouter and includes the network. Strands Decider ran on the same laptop, one request at a time.
The harness is on GitHub: https://github.com/sanaiqbalw/jev-101-tests.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article