The Witness That Could Only Nod: a Week With Strands Decider, Part 2

Thursday morning. I took the witness out of the control room and put it on the door.

 

By then I'd stopped arguing with it. (Part 1  if you missed the first three days .) Three days of questioning and every disagreement had turned out to be mine: a weak baseline, a leading question, a scorer that read "by 5 pm" as five. It had picked tools as well as Nova Lite, checked arguments better, run my agent for a sixth of the price without a single stall, and told me how sure it was every time, honestly, in both directions. The one place the number hadn't warned me was Tamil, and even there it hadn't lied; it simply had no way of knowing.

 

So the four lines of questioning that were left weren't about whether I could trust the number any more. I could. They were about what the number was worth on jobs where pointing isn't enough, where the witness has to have read something. The door was the first, and the one I'd been saving.

 

 

 

Eight hundred visitors. Prompt injections from the deepset and Gandalf datasets, harmful requests from JailbreakBench, and matched honest traffic, each arriving five ways: plain, in leetspeak, base64 with a hint, hidden inside a tool result, split across two turns. Three guards on the same door: the witness with a pair of yes/no questions, Nova Lite with the same questions, and the managed guard already standing there, Amazon Bedrock Guardrails with every input filter at HIGH including prompt-attack detection.

 

 

Plain text first, at a threshold of 0.5. The witness stopped 63% of attacks and bothered 7% of honest visitors. Nova Lite 74% and 17%. Guardrails 73% and 12%. Then the thing a probability lets you do that a verdict doesn't. I slid the witness's threshold until it troubled honest visitors exactly as often as Nova did, 17%, and its recall went to 92.5% against Nova's 74%. The LLM gives you an answer. The witness gives you a dial, and on plain text the dial is the best of the three.

Then the disguises, and all three guards turned out to be watching a different gap.

 

 

 

Read it by column. Guardrails sees inside tool results, which the witness doesn't, and misses spelling tricks, which the witness half-handles. The witness half-reads spelling and doesn't see inside nesting. Nova is middling at both and stops anything in base64 on sight. A single-pass pointer head doesn't look inside nested or encoded text; the LLM sometimes does, slowly; the managed guard was built for the tool-output case. None of this is a failure of the witness. It's the shape of the job.

 

One guard on a door has that guard's gap, whoever the guard is, and the right answer is the one every security person already knows: layers. The witness is the fast, cheap, honest first layer for plain text. Guardrails takes the tool output. An LLM or a decoder takes anything wrapped.

 

And now the oath, the sentence I'd started the week stuck on. Above 0.9, 96% of what the witness stopped was a real attack. Honest, again. But only 8% to 19% of attacks ever reached 0.9. So the rule from the README is true and, at this particular door, not much use, because almost nothing clears the bar. The confidence wasn't wrong. It just wasn't high, and that is the number telling you, correctly, that this isn't the desk it's best at.

 

That was the moment I found the sentence for the whole week. Calibrated is not capable. And the witness had never pretended otherwise.

 

One more thing from the door, and it rhymes with Part 1. I asked the same question three ways. Two long yes/nos with the criteria written out: 63%. Two blunt ones, "Is this a prompt injection?": 32%. One three-way choice between honest, injection and harmful: 61%, at half the latency. Half the recall vanished on wording alone. On Tuesday the phrasing had chosen which way the witness erred; on Thursday it chose how much. And it was weaker on the German items than the English ones, 16% to 28%: the second time in a week the oath turned out to have a home language.

 

Model routing is the desk everyone wants filled. Send the easy prompts to Nova Micro, the hard ones to Pro, pocket the difference. I had every Nova tier answer the same 200 MMLU and GSM8K prompts, so I knew which tier each one truly needed, then asked three triage nurses to pick a tier from the prompt alone.

 

None of them could. Micro was right 86.5% of the time at a thirteenth of Pro's price, so the ideal nurse says "Micro" almost always and "Pro" for eighteen specific prompts. The witness sent 146 of 200 to Lite. Nova Lite as nurse sent 163 to Lite. The witness's score question rated every single prompt "medium". Asked to predict which prompts Micro would get wrong, its probability scored an AUC of 0.51. A coin. Looking hard and being hard are not the same thing, and no model read the difference off the text.

 

Here is why this desk made me trust it more, not less. Its median confidence on routing was 0.32, and not one of 200 decisions reached 0.9. Under the escalation rule from Part 1, every single routing decision would have gone to the LLM, which couldn't do it either, but at least the witness had said so. On the door it had been sure about the easy cases and quiet about the rest. Here it was quiet about everything, and right to be. Both times the number told the truth. A colleague who says "I don't know" on the one job nobody can do is a colleague you keep.

 

The score result recurs, so it gets its own line. The answer is an expected value over the levels, so uncertainty pulls it towards the centre, and rounding it to a tier always gives you the middle one. score is an estimator whose ranking you can trust and whose absolute values you shouldn't. It makes a poor classifier. As the next desk shows, it makes a decent judge.

 

The first bet I'd have lost.

 

The MT-Bench human judgements are 3,355 pairs of chatbot responses with a human's preference attached, and the literature on LLM judges, Zheng et al. (2023), is mostly about how badly they're swayed by which answer came first and which was longer. So I had the witness judge 300 pairs twice each, swapping the order, and counted a verdict only when it survived the swap.

 

It agreed with the humans on 79.8% of the pairs it was consistent on. Nova Pro, on 150 of the same pairs, 76.3%. Zheng et al. report GPT-4 reaching "over 80%" agreement with humans on these pairs, about the rate at which humans agree with each other. A model that cannot write a sentence was judging sentences at roughly the level people agree with people, and a frontier-tier Nova wasn't doing better. It leaned to whichever answer came first, 55% against an unbiased 50% (Nova 49%), and flipped its pick on the swap 14% of the time, so judge twice, swapped, with either of them. Neither rewarded length beyond what the human labellers did.

 

 

 

I'd expected a model with no language head to have no taste at all, and I don't have a tidy explanation for why it isn't. The pointer head is doing something with the Qwen representations underneath that behaves like preference, at 836 ms a pair against Nova Pro's 1,296. It works. I checked it the way the literature says to check it, and it works.

 

The quiet win was score again. I had it rate each response alone on a seven-level rubric and compared the two numbers, and the ordering agreed with the humans 78.6% of the time, nearly as well as the head-to-head question. The values themselves huddled between 1.8 and 4.5 on a scale of 0 to 6. Ranking, trustworthy. Absolute values, not. Same lesson as triage, now in its favour.

 

The second bet, and the desk I'd planned to skip. I sat the witness at it on the last afternoon for a minimal shift: 200 synthetic six-turn conversations in which the user states three facts and the assistant volunteers one of its own, then a candidate memory line to file or reject. Half the lines are things the user said. The other half are plausible and false in one of three ways: the right kind of fact with the wrong value, a fact the assistant said rather than the user, or a kind of fact that never came up at all. One question: did the user state this?

 

 

It never dropped a fact the user had stated, and never once put the assistant's words in the user's mouth. Nova Lite forgot 15% of what was said. The two have opposite temperaments: the witness's "yes" is strong and its "no" is weak, mean probability 0.84 against 0.42; Nova is careful to the point of amnesia.

 

Its one edge is a specific one. When the candidate line was a kind of fact the conversation had never touched, it filed it 21% of the time. The user had talked about their city, their job and their pet, and the witness agreed they had said they were allergic to shellfish. It is good at who said this and poorer at was this said at all, and the second matters for a memory system, because that is how a memory invents. The fix is cheap and mechanical: an exact-match check for fact types the conversation never mentioned, before the question is asked.

 

 

 

Everything in that table is on templated sentences from a generator I wrote in an hour. It's the reason to run the real test, not the result. But it was the desk where the model I'd written off beat the incumbent on every line but one, and it was the last thing I ran before I closed the laptop on Tuesday night.

 

Context management, the seventh job on the launch post, I read as reranking: given a query and twenty retrieved passages, put them in order. SciFact from BEIR, BM25 top-20, the witness's score on a four-level relevance rubric, against the BM25 order and bge-reranker-base, a 278-million-parameter cross-encoder built for exactly this.

 

 

 

The witness beat BM25, nDCG@10 0.676 against 0.642, and lost to the cross-encoder a seventh its size, 0.720, at 6.6 seconds a query, because every passage is a full forward pass and nothing batches. That isn't a mark against it. Reranking is not what a pointer head is for, the same way a judge isn't a filing clerk. I took it off that desk the same afternoon and don't hold it against it.

 

 

 

The last thing I built is the first thing I'd hand you. Twenty system prompts from five agent shapes, each hand-labelled generate, decide or extract, put to the witness as a choice. It got 17 of 20, and the three it missed were the three it was least sure of: 0.41, 0.28 and 0.16, against a median of 0.80 when right. Set the line at 0.5 and it labels fourteen call sites on its own, all correctly, and hands you six to check, three of which it would have got wrong. python -m gauntlet.audit --file your_sites.json is in the repo. Paste your agent's prompts in and it tells you which of your calls are decisions, and which it isn't sure about. It's the witness doing to your agent what I spent the week doing to it.

 

 

 

Six desks out of nine, and on the two it turned down it was the one who told me.

 

 

I started the week stuck on at 0.9 it's right 95% of the time, wanting to know on whose tasks. The answer is: on mine. Everywhere I checked, including the desks that weren't its job, the number meant what it said. That is the thing I hadn't understood about a calibrated decision model before I had one in the room. The number it holds up doesn't tell you it's good at the job. It tells you, per decision, whether it's sure, and it is honest about that in both directions. On tools and paperwork it was sure and right. On triage it was unsure and right to be. On the door it was sure about the easy cases and quiet about everything else, and the quiet was the useful part. Asked to grade its own agent's answers, it never got confident at all, and that was the most valuable thing it told me all week.

 

A word on scale before the sentence I'd write. Everything here was measured on one Mac, on a few hundred to a few thousand items per desk, with datasets I chose. That's enough to say what the model did on my tasks and not enough to say what it can't do on yours, which is why every row is in the repo and every table regenerates with one script.

 

 

 

So here is what I'd write under the oath, and what I've written into my own agents since. Let the small model decide when it's sure, and let the number tell you when it isn't. That rule ran my agent for a sixth of the price with no accuracy lost and no stalls. It would also have sent every triage decision and most of the door back to the LLM, which is exactly where they belonged, and it would have done it without my having to know in advance.

 

Calibrated is not capable. But a witness that tells you which is which is the only kind worth calling, and this one is.