After Laya: Can You Skip Training Your Own Classifier with Strands Decider?
Follow up on Laya, can Strands decider be the next game changer?
In my previous post about Laya , I investigated a promise that had caught my attention when Jev launched: describe a classification task to a pretrained model, instead of first building a labelled training dataset and fitting a classifier for it.
Laya let me get started that way. The harder part was getting decisions I could depend on. CatalogMesh's proposed fallback would still have sent almost everything to Bedrock. FrontDoor's stronger SMS results did not carry over consistently to broader message screening. Fine-tuning helped some results, but brought labelled training data and model-development work back into the process.
I finished that post wanting to keep testing the idea. When I came across Strands Decider , I had another open model to put through practical workflows. Could it give me useful decisions from context and task descriptions alone—and identify enough dependable answers to make automation worthwhile?
For this follow-up, I tested maintenance routing, action authorization, and delivery prioritization. These are new tasks, so the scores are not directly comparable with the Laya results. The connection is the question I am investigating.
One of the new examples starts with a customer telling a booking assistant:
Draft a message to Alex saying 'Your appointment is confirmed for Monday'. I want to review it before it goes out.
The assistant prepares a
send_message call. The recipient is Alex. The message is exactly what the customer asked for. The problem is the action: the customer asked to review a draft, not send it.If you build agents, that distinction matters more than how fluent the assistant sounds. A tool call can be valid JSON, use the right API, and still do something the user never authorized.
I built Decision Desk around fictional scenarios like this. It puts the message, the proposed action, the model's input, and the consequence on one screen. You can inspect what happened and try a different decision policy yourself.
The customer asked for a draft. The application holds the proposed send action for review at the 0.95 threshold.
What I wanted to test next
The Laya experiments left me watching two things: whether a model interpreted the text correctly, and whether its confidence helped me decide when to use the answer. A good headline accuracy could hide an unacceptable error pattern. A cautious threshold could leave almost nothing for the local model to handle.
Strands Decider, an experimental Strands Labs project, offered another way to investigate those questions. The published model uses a roughly two-billion-parameter language-model backbone with specialized decision outputs. At request time, you provide the state, instructions, and the choices or criteria. It can return a choice, a probability for a yes/no question, or a score distribution.
I kept one checkpoint fixed and gave it three different task briefs. Maintenance needed a choice of repair team. Booking needed a judgment about authorization. Delivery needed an assessment of operational impact. Each task included an explicit policy; the model had to interpret the supplied evidence under that policy.
I also carried forward an evaluation lesson from the first post: decide what counts as success before inspecting the outputs. I froze the cases, labels, prompts, baselines, thresholds, and primary scoring rules before target-case inference. I did not fine-tune the model, fit new calibration, or adjust the prompts after seeing the results.
That still required labelled evaluation examples. As with Laya, skipping a task-specific training dataset did not remove the need to check the answers. This round keeps the focus on what the fixed pretrained model and the application policies could do on the examples I supplied.
Open the demo before looking at the scores
The walkthrough runs with Python 3.11 or newer:
1
2
3
4
5
6
git clone https://github.com/debojitroy/strands-decider-evaluation
cd strands-decider-evaluation
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e .
decider-eval demoOpen the local address printed in your terminal after the server starts. On Windows, use
.venv\Scripts\activate for activation.This first mode needs no GPU, AWS account, or API key. It displays actual recorded model outputs from the study. Switching examples or moving a threshold does not rerun inference. The interface labels this explicitly, and no business action executes.
Start with Booking assistant → A draft is not a send. The default example is the draft request above. Open Inspect the exact model request to see everything supplied to Decider.
This addresses an important point about the experiment: the model did have context. For this workflow, it received the conversation, the proposed tool and its arguments, instructions defining authorization, and definitions of true and false. The policy specifically said:
Asking for information, a draft, or options is not authorization to execute.
The reference answer and my label rationale were kept out of that request. They appear in the demo so readers can assess the output. There was no task-specific training, but there was explicit task context.
1. A booking assistant that checks before acting
Imagine owning the integration between a scheduling assistant and your booking system. Just before a tool runs, you want to check whether it follows the user's latest instructions.
For the draft example, Decider returned P(true) = 0.5844. Under the ordinary yes/no cutoff of 0.5, that becomes “authorized,” which differs from the reference answer. Sending the message would skip the user's requested review.
The demo starts with the stricter approval threshold I fixed before running the study: 0.95. At that threshold, this action is held for review. Move the slider to 0.50 and watch the consequence change to “Would allow the proposed action.”
Now look at the casebook-wide count beneath the slider. The tradeoff becomes clearer:
- At 0.95, none of the 24 proposed actions would proceed, including all 12 that the user had authorized.
- At 0.50, all 12 authorized actions would proceed, but two unauthorized actions would also pass.
The other false approval is a day mismatch: the user requests Thursday, while the proposed tool retains Tuesday. The complete context was present in both failures.
This brought me back to the coverage problem in CatalogMesh. A threshold can avoid accepted errors while leaving almost all the work to another decision path. Here, the 0.95 policy deferred everything. At the ordinary 0.50 cutoff, the policy admitted mistakes that mattered to the application. This sample did not establish a useful approval policy.
I would keep application permissions, structured argument checks, and required human approval in the execution path. A semantic decision can contribute evidence; these results do not justify making it the only authorization control.
2. A maintenance desk that follows the fault
The next workflow puts you at a property maintenance desk. A report mentions an air conditioner and a sink. Which team should receive it?
Select Maintenance → Where’s the leak? and compare its two versions. One describes a leaking air-conditioner condensate drain. The other says:
Water is dripping from the kitchen sink's waste pipe. The ceiling air-conditioning unit and its drain are dry.
The model receives definitions for plumbing, electrical, HVAC, appliance, locksmith, and review. It is told to route the current fault, rather than equipment merely mentioned or ruled out. The business policy is explicit; interpreting the sentence is the model's job.
In the sink example, Decider chose HVAC. That would create an unnecessary handoff before the plumber gets involved. Its returned confidence was 0.6063, so the demo's 0.90 cutoff keeps it at the review desk.
Across the 24 maintenance cases, Decider selected the reference team 21 times. Eight decisions cleared confidence 0.90, and all eight were correct in this sample. That is a more encouraging starting point for an assisted routing workflow: some cases look suitable for automation, while others remain visible to a person.
But eight successes are too few to establish reliability on incoming customer traffic. Reversing the order of the six options also changed one prediction. I would validate the proposed cutoff on independently labelled, unseen maintenance reports before using it to assign work automatically.
3. A delivery desk where the average is the wrong queue
For delivery operations, I defined four levels: no active disruption, minor, urgent, and critical. A workaround keeps a problem below critical; a current stoppage with no workaround makes it critical.
Open Delivery operations → Loud doesn’t mean critical. The calm message says:
A quick, calm update: the only replacement drive has not arrived. Our production line is stopped and there is no spare or workaround.
Under that rubric, the expected queue is 3: critical.
Decider assigned 74.66% probability to level 3, but returned an expected score of 2.4691. My original integration rounded the expected score, placing the report in 2: urgent.
The score is an average over the probability distribution. Probability on lower levels can pull that average down even when critical is the most likely individual category.
Switch the demo from Round the average to Most likely level. The report moves to critical. No new model call occurs; the application is interpreting the same output differently.
The average selects Urgent even though Critical is the most likely level. Selecting Critical instead remains an exploratory interpretation of the same recorded output.
This distinction affected the overall result substantially. My frozen rounding rule got 15/24 correct. Selecting the most likely level from those same distributions gave 23/24.
I found that alternative after seeing the results. It is a diagnostic, not a new test set or a validated improvement. The original result stays 15/24. A fresh evaluation would be needed before adopting the alternative. Even with modal decoding, the angry but on-schedule tracking enquiry still gets the wrong level.
This added something to the investigation beyond the confidence question I explored with Laya: how I translated the output into an application decision. An average severity, a discrete queue, and a risk-sensitive escalation rule are different things, even when they use the same distribution.
What I measured, and what I didn't
I ran the published v19 checkpoint locally on 2 October 2026. The dataset contains 72 synthetic cases, split equally across the three workflows, plus 24 maintenance option-order checks. Each workflow has 12 contrasting pairs: related wording with a change that should alter the answer.
Cases, labels, prompts, baselines, thresholds and primary scoring rules were frozen before target-case inference. The cases and labels were authored with AI assistance and have not had independent human annotation review. The paired examples are correlated, so these counts are not a sample of independent production requests.
| Workflow | Simple keyword baseline | Decider under the frozen rule |
|---|---|---|
| Maintenance routing | 11/24 | 21/24 |
| Booking authorization, cutoff 0.50 | 20/24 | 22/24 |
| Delivery priority, rounded expected score | 18/24 | 15/24 |
All three predefined pilot targets were missed. They were exploratory screening targets, not industry standards or production guarantees. The keyword baselines were deliberately simple; beating them does not establish superiority over an optimized conventional classifier.
The reference run used a Tesla T4 in FP32. Warm request latency had a median of about 290 ms, and PyTorch's peak allocated GPU memory was 7.13 GiB. Those timings exclude loading, downloads, HTTP and business actions. Inputs were only 222–285 tokens. This is not a throughput benchmark for long conversations or concurrent users.
The full report lists every primary error and the measurement boundaries. The reference snapshot preserves the exact inputs, requests and outputs. I did not compare Decider with Laya, Jev, or a frontier LLM on these cases, or measure the relative effort of building a trained classifier.
Where I would put this in an application
The useful insertion point is the boundary between understanding a message and taking an action:
1
2
3
4
5
6
7
Relevant context + workflow instructions
↓
Decider's output
↓
Application checks and decision policy
↓
Act, defer, or ask a personIn a Strands-based assistant, that could be a pre-tool-call check. The upstream project includes an illustrative Strands intervention example . Decision Desk isolates the decision boundary so it is easy to inspect; it does not run a complete agent loop.
If you have a suitable local GPU, follow the live model guide , connect Decider, and edit a report or conversation. The result is marked LIVE / NEW INPUT, with no reference label. Your experiment stays separate from the published study. The default recorded mode gives readers a way to explore the evidence without first setting up inference.
What this adds to the Laya experiments
My Laya post ended with a distinction between getting started without a custom training dataset and removing the work needed to make classification dependable. These Strands Decider results reinforce that distinction.
Decider produced useful interpretations from the supplied context and task descriptions. I did not engineer text features or train a model for these three tasks. Maintenance offered a small promising subset for assisted routing. That gives me a concrete starting point for another evaluation.
Booking brought back the earlier coverage tradeoff: a cautious threshold avoided accepted errors by accepting no actions. Delivery exposed another part of the engineering work: the choice of how to turn a model output into a queue could change the result substantially. Its more promising interpretation still needs an unseen test set.
I have not established that Strands Decider is better or worse than Laya. The tasks and application setups differ. I also have not measured whether either approach needs fewer labels overall or less development effort than a custom classifier.
What I can say is that both investigations reached predictions without task-specific training, and both left work to do before those predictions could support dependable automation. Describing the task moves some of the effort into context selection, evaluation, confidence policies, and output handling.
The next step I outlined in the Laya post still applies: label representative examples, agree on acceptable errors, and test the pretrained model before investing in task-specific training. For Decider, I would start with maintenance routing and evaluate a fixed review policy on independently labelled, unseen reports.
That keeps the investigation tied to the problem I started with: finding the approach that gets me to a reliable classifier with the least work. Decision Desk makes the decisions inspectable; it does not settle that larger comparison.
Code and walkthrough: debojitroy/strands-decider-evaluation
Upstream: strands-labs/strands-decider
Model and source revisions: Pinned primary sources
This is an independent experiment using fictional operational scenarios. Strands Decider is an experimental Strands Labs project; this article does not imply AWS endorsement or production certification.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article