The Witness That Could Only Nod: a Week With Strands Decider, Part 1

The agent had gone quiet.

 

Not finished. Quiet. There's a difference, and you learn it on a Sunday afternoon with sixty tasks queued and a cup of tea going cold. Sixty tasks, a reference agent built the way the Strands samples build theirs, orchestrator on Nova Lite, three Nova Micro specialists wired in as tools, a verifier, a review gate, a final answer, and somewhere past the halfway mark one task went in and nothing came out. No error. No timeout. The cursor blinked. I checked the Bedrock console; tokens were still flowing. Something in there was alive and busy and not telling me what it was doing.

 

 

Three-quarters of an hour later it died, and left this:

 

 

The orchestrator had picked a tool, read the result, picked a tool, read the result, picked a tool, and kept going until the SDK stopped counting. When I went through the rows afterwards, that was only the loudest of three ways it had failed. Fourteen tasks out of 180 never finished: two recursed until Python gave up, five produced a tool call Bedrock refused to parse, and seven simply talked until they hit the token limit and were cut off mid-sentence, like a man being escorted from a wedding speech. Three different ways of generating badly. Twice it took most of an hour to do one of them.

 

 

 

That was the colleague I was about to replace. The replacement, Strands Decider 2B, couldn't speak.

Strands Labs released it on a Thursday. Strands Decider 2B: a two-billion-parameter model with the language head cut off. It cannot write a sentence. It cannot explain, argue, apologise or stall. You put a situation in front of it and a set of options, and it points at one and holds up a number for how sure it is. One forward pass. About a hundred milliseconds. Then it waits for the next question.

 

 

 

Underneath, it's Qwen 3.5-2B with the language-model head removed. Where the head was, there's a pointer head of about a million parameters that scores each option you supply against a single answer position; a softmax over those scores is the whole of what it says. You can question it three ways. noul: yes or no, with a probability. choice: one of up to 255 options, with a probability for each. score: place the input on an ordered scale, get back an expected value. There's no hosted version. pip install strands-decider, pull the weights, and it's sitting in your room in three and a half seconds. It can hold 4,096 tokens in its head, and a question may take three-quarters of that.

 

Strands calls the arrangement a hybrid agent: a frontier model generates, a small local model decides. TypeSafe's Jev opened the category a few weeks earlier, OpenAI has a hosted cousin in its Decisions API. What this one adds is that you can download it, training data and scripts included, and keep it.

 

And it arrived under oath. In the README: at a confidence of 0.9 or more it is right about 95% of the time.

 

Here is what unsettled me about that sentence, and it's the reason this piece exists. A witness that can only nod cannot be cross-examined. You cannot ask it why. You cannot catch it contradicting itself, because it never says anything long enough to contradict. All you get is where it points and the number it holds up. If the number is honest, it's worth more than any colleague I've worked with, because it tells you when not to trust it. If the number is not honest, it is the most dangerous component you could put inside an agent: wrong, at a hundred milliseconds a time, with a perfectly steady hand.

 

The oath was sworn on its previous employer's benchmark, on its previous employer's GPU. I wanted to know whether it held in my room, on my questions, with my colleagues sitting beside it for comparison. So I built the room. Seven lines of questioning, one for each job the launch post said it was for. Nova Micro, Lite or Pro in the other chair every time, and for one line the managed guard already on the door. Every answer kept.

 

I'll tell you now that by the end of the week I had stopped trusting one thing in that room. It wasn't the witness.

 

I asked it something small. 70 ms. The post had said 153 ms on an M3 Pro, so either my chip is newer or they were being modest.

 

 

Then I made the question longer and found the first thing about it that was simply true. It reads everything you hand it, at about 0.43 ms a token, and nothing is free. A full 2,874-token situation takes 1.23 seconds. Options are text too, so a choice across a hundred described tools costs 774 ms, and at two hundred it refuses outright. Not silently. The prompt was truncated through its option list. The schema says 255 options. The budget says fewer, and the witness tells you when you've hit it, which is more than the colleague on the Sunday did.

 

Ten identical questions, ten identical answers, to the last decimal. Five questions about one situation in a single call: 235 ms, against 350 for five calls, a third saved and no more. And the same question on my Mac's CPU instead of its GPU took 14.3 seconds, because two of Qwen 3.5's attention kernels only exist for CUDA and the fallback is reference PyTorch. "Runs on CPU" is true in the sense that it finishes.

 

 

 

Tool selection is the first job on the launch post, so I pressed where it would hurt: the length of the list. For each item in the Berkeley Function-Calling Leaderboard I gave the witness the right function and k wrong ones, shuffled, and asked which to call. k from 5 to 200.

 

 

 

The first evening, I led the witness. I had Nova Lite pick a tool by writing its name in a constrained text prompt, and the witness beat it by 8 to 14 points at every list length, and I went to bed with a headline. In the morning I asked the colleague properly, passing the functions to Nova as real Bedrock tools with toolChoice: any and reading the toolUse block, and the headline was gone. 99.5% against 99.5% at five tools. 88.9% against 88.4% at a hundred. Disagreements split down the middle. Nova Micro sat four points lower at fifty and a hundred. A model that can't write had matched a hosted LLM with zero training data. Equal. Not better. The leading question stays in the transcript because it's the first lesson of the week: choose the easy colleague and the witness looks like a star.

 

What it adds is in the margins. Nova Lite's tool-use call failed 5.6% of the time with Model produced invalid sequence as part of ToolUse; count those as wrong and it's at 84% with a hundred tools. The witness's answer is a softmax over the options you handed it, so a malformed answer isn't a thing it can produce, and across 2,300 questions it never once failed to point. And the oath held. At 0.9 and above it was right 99.8% of the time with five options and 94.9% with two hundred. Off its own benchmark, on my questions, the number was if anything modest.

 

Two marks against it. Its latency grows with the list and Bedrock's barely moves, so past about fifty described tools Nova is faster from a laptop. And in long lists it stops reading to the end: with 100 options the right one sitting in the last quarter costs seven points; with 200, twenty. Nothing was truncated. The pointer head weights options far from the question less. Shuffle, or keep the list short.

 

There was a third question on this line, one I'd written on the first day and hadn't asked. It was in a language I hadn't used yet. I left it where it was.

 

Argument grounding is the launch post's own worked example: before a tool runs, ask whether its argument values trace back to something the user actually said. I built 400 synthetic conversations. The user states values. A tool result suggests different ones. Filler turns push the user's words further back. Then a call is proposed with one value either faithful or swapped.

 

 

 

Witness 93.5%. Nova Lite 91.0%. 229 ms against 605. It accepted 98% of good calls and caught 89% of bad ones, and here its number was the sharpest of the week: not one call below 0.4 was grounded, every call above 0.7 was, and the 18% in between were precisely where the uncertainty lived. Escalate that band and the rest is essentially always right.

 

The edge is dates. An off-by-one-day swap was caught 56% of the time, against 97% to 100% for names and order numbers. It sees "2026-11-14" and "2026-11-15" as the same thing. Put an exact-match check for dates and amounts in front of it. And I'd guessed wrong about what would hurt. I expected values lifted from the tool result to be the hard case, because the string is right there in the context and the question is about who said it. They were caught 87% of the time against 93% for invented values. Distance mattered more: ten turns back cost fifteen points.

 

Then I asked the same 400 questions three ways, and this is the finding I'd want you to carry out of the room if you carry one.

 

The launch post's careful wording: 93.5%. A blunt "Are the arguments correct?": 94.0%, a shade better. A strict "Is every value stated verbatim in a USER turn?": 81.8%, and inside that number a shape the accuracy hides. The strict wording caught 100% of bad calls and rejected 36% of good ones. The blunt one accepted every good call and let 12% of bad ones through.

Same witness. Same evidence. Same accuracy within a point for the first two, and opposite mistakes for the third. The phrasing hadn't just moved how often it was right. It had chosen which way it would be wrong. In a courtroom that's leading the witness, and it's counsel's fault. I was counsel. Write the question for the mistake you can afford, because you are choosing it either way.

 

Back to the Sunday, and the cold tea.

 

 

 

Six LLM calls per task in that agent. Three of them are decisions: which specialist, is this answer complete, does this need a human. I gave those three to the witness, with one rule: if its number fell below 0.6, the LLM it was replacing would answer instead. Then I ran the same 180 auto-scored tasks both ways, three seeds each.

 

 

 

A sixth of the cost. A third off the median, the tail halved, every run. Accuracy up 3.7 points, which I won't claim: on sixty tasks a run swings fifteen either way, so the word is unchanged. The witness deferred to the LLM on about a third of its decisions, so these are the numbers with escalation on, which is how anyone would run it.

 

And the top row. Every task finished. No stalls. An orchestrator that chooses tools by generating text can walk into a loop, or run out of tokens, or produce a tool call nobody can parse. A softmax over three specialists can do none of those; there is no text to run out of and nothing to parse. The witness didn't finish because it was cleverer. It finished because all three things that happened on Sunday cannot happen to it.

 

One of the three decisions was the review gate, and it handed me the strangest number of the week. The LLM gate sent 97% of answers to a human. The witness sent about half. One of them was wrong about what deserved a human's eyes, and the auto-scorer couldn't tell me which.

 

 

So I read the evidence myself. Forty-six answers where the two gates had disagreed, graded blind, with the gate decisions hidden from me: wrong; right but messy, leaked <thinking> tags, raw LaTeX, half a sentence; right and clean.

On the question the gate had actually been asked, is this answer likely correct, the witness agreed with my grades 87% of the time and the LLM 17%, because the LLM had flagged everything and most of everything was fine. On a question nobody had asked, is this answer fit to show a user, both were near chance, 43% and 39%, because the witness had waved the messy ones through. It had answered exactly the question put to it and nothing beside it.

 

So I put the other question. Two yes/nos on the same 46 answers: is it numerically correct, is it cleanly formatted. It caught one wrong answer in seven and one messy answer in fourteen, and its number never left the band between 0.25 and 0.80. It cannot judge a generated answer by reading the transcript. And it never once claimed it could; it just never got sure. The 87% had mostly been the virtue of not crying wolf.

 

That should have closed the line. It didn't. Because while I was grading those 46 answers, four of them wouldn't sit still.

 

Four answers I had marked right and clean were recorded in the results as wrong.

 

I read them again. Then a third time. One ended: Fred must deliver 160 red roses by 5 pm. The correct answer was 160. The scorer had recorded 5.

 

The scorer took the last number in the text.

 

 

I sat with that for a while, because of what it meant. Every maths accuracy in the pack, every run, every table I'd built for three days, had been computed that way. And the bias wasn't random. The agent's last step restates the answer for a user, in prose, and prose ends in context. By 5 pm. Four years from now. An additional 2,000 miles. The LLM plane's answers ended in a bare number more often, so they'd been marked right more often. Since Sunday I had been quietly penalising the witness for writing like a person.

 

 

 

I fixed the scorer and re-marked every stored answer, no new model calls, nothing re-run. Fourteen answers flipped from wrong to right. None flipped the other way. All fourteen were the witness's. The table above is the corrected one. I had been a day from publishing its accuracy as 82.2%, and I'd have been true to the rows and wrong about the agent, which is the worst kind of wrong because it comes with a spreadsheet.

 

The witness had never lied to me. The transcript had. Everything I'd believed for three days about which gate was crying wolf, which plane was more accurate, how wide the gap was, I had read off a record with a hole in it, and the only reason I found the hole was that I sat down on a Tuesday night and graded 46 answers by hand. I said at the start that by the end of the week I'd stopped trusting one thing in the room. It was my own scorer. Check it before you believe anything else in here, mine included.

 

The one from the first day. The one in a language I hadn't used.

 

I took a hundred of the tool-selection questions, translated them into Hindi and Tamil with Nova Pro, left the tool names in English, and put them to the witness again at k=20. The numbers it held up looked like the numbers it had held up all week.

 

English, 95%. Hindi, 90%.

Tamil, 63%.

 

 

Thirty-two points. Same items. Same tools. Same steady hand. Machine translation is a caveat, but Hindi went through the same translator and lost five. What's showing through the pointer head is the pretraining of the model underneath it, and the witness was never taught language at all, so it has no way of telling you this is happening. On the Tamil items its confidence looked the way it had looked on the English ones. This is the one place in the week where the number didn't warn me, and it's the one that matters most to me personally. I don't read it as the model failing in Tamil; a hundred machine-translated items on one Mac is not that kind of evidence. I read it as the one check the confidence can't do for you. If you build agents for Indian languages, run yours through this before you trust anything in this series. Including me.

 

 

Before anyone else could question it, the witness needed an HTTP server. The bundled one handled one client at 11 requests a second and died at two. Not slowed. Exited, on the first pair of concurrent requests, and the log said why:

 

 

Metal will not let two threads encode to one device at once, and the server is a single uvicorn worker with nothing serialising access. I wrote the same server with one threading.Lock around the forward pass, and it took 1, 2, 4, 10 and 25 concurrent clients at a flat 16 requests a second, client latency equal to concurrency times 62 ms to within a few milliseconds. Sixteen decisions a second from one Mac, predictable queueing, and past that more Macs or a GPU. I tried for the GPU number on a g6 and lost an afternoon to instance capacity and a conda-activation quirk in the Deep Learning AMI. It's the follow-up, and I'd rather say so than guess.

 

 

 

Three lines of questioning in, the pattern was clear, and it wasn't the one I'd braced for.

 

Every time the witness and I disagreed, I went back and found the fault on my side of the table. I'd led it with a weak baseline. I'd phrased the question so it would err the way I'd written it to err. I'd marked its answers against a transcript that said 5 where it had said 160. The number it held up had been honest every time. Where it couldn't do the job, it told me the only way it can, by never getting confident. Where it could, it was right at 0.9 as often as it had sworn it would be.

Except once. In Tamil, the number looked the same and the answers weren't. The oath has a jurisdiction, and it is the language the model underneath was raised in. That isn't a verdict on the model. It's a test to run before you trust it in yours.

 

On the Thursday I took it out of the control room and put it on the door, beside the guard already standing there, and sent eight hundred attackers at both of them in five disguises. That's Part 2. I'll say two things about it and no more. The oath I started with turns out to be true and useless in the same sentence. And I lost two bets I'd have put money on, in opposite directions, to a model with no language head.

 

 

 

The room is at github.com/Felina14/decider-gauntlet . Every table regenerates from the rows with one script, and the adapters are written so you can put Jev, the Decisions API, or Decider v20 in the same chair when they arrive.