A Physical AI Robot Episode Judge That Knows When It Does Not Know: Strands Decider in Strands Robots
I added Strands Decider as an opt-in episode-judge backend in strands-labs/robots. It labels a recorded robot episode with one HTTP request (about 1.8 s), returns calibrated probabilities, and writes a label only when it is confident. On a 12-episode MuJoCo test it wrote zero wrong labels and left every unsure episode, with its probabilities, for the existing VLM judge.
Strands Robots records robot episodes as LeRobot datasets and labels them for training: was the execution smooth, what went wrong, did the task succeed. Today that labelling runs through create_judge_agent, a vision-language agent that calls four tools per episode. I built a second, opt-in backend on top of Strands Decider, the small typed-decision model from the Strands team. It asks the same three questions in one HTTP request, gets calibrated probabilities back, and writes a label only when every answer clears a confidence threshold. Anything it is unsure of goes to the agent judge or a person, with its probabilities attached.
I tested it against the official Strands Decider v19 and v21 releases on a 12-episode MuJoCo ladder whose ground truth I set by construction. Both models were at chance on this footage, and both knew it: every answer was low-confidence, so the backend deferred all 12 episodes and wrote no wrong label, at about 1.8 seconds per episode. The plumbing, the safety contract and the abstention work against the real model today. A decider trained on robot episodes is what turns the deferrals into labels.
The proposal is issue #4572 and draft pull request #4573 in strands-labs/robots.
Alt text: Diagram: recorded episodes flow into DeciderJudge, which sends four frames and three typed questions in one request; confident answers are written with write_label, unsure ones are deferred to create_judge_agent or a person.
How the backend sits in front of the existing agent judge. Nothing changes unless a caller constructs it.
The use case: labelling thousands of robot episodes
A robot learning loop produces a lot of recordings. Every policy rollout, teleoperation session and simulated run becomes an episode, and an episode is only useful for training once someone says what happened in it. Strands Robots splits that into two layers. A deterministic predicate decides success or failure from the robot state, and that verdict is authoritative. A judge then adds annotations the predicate cannot compute: a quality grade, a failure mode, and its own success opinion.
All three judge outputs are closed sets. Quality is one of low, medium or high. Failure mode is one of eight tags (jerky_motion, near_miss, camera_occlusion, wrong_but_lucky, drift, collision, incomplete, other) or none. The success opinion is yes or no. An agent loop spends most of its time deciding which tool to call next, while the judgement itself is a pick from a short list. That is exactly the kind of question Strands Decider answers: a typed decision (choice, yes/no or score) with a probability for every option.
The second reason is confidence. The agent judge writes an unsure label the same way it writes a sure one, and the only check is measure_agreement against a human holdout afterwards. A decider returns a probability for every option, so the backend can decline to write before the label exists.
How it works
DeciderJudge talks to a running Strands Decider vision server over plain HTTP. The robots side adds no dependency; the client is the Python standard library, and torch and the model live in the decider's own environment.
1
2
3
pip install "strands-decider[vision] @ git+https://github.com/strands-labs/strands-decider@3e94e9d"
strands-decider serve StrandsAgents/strands-decider-2B-hobson-v19 --vision \
--model-name strands-decider-2B-hobson-v19@bb282d7 --port 8765For each episode the backend does five things.
- It reads the deterministic verdict, only to refuse an episode that has none. The verdict is never sent to the model, so the model cannot copy it.
- It samples four evenly spaced camera frames with the existing sample_frames tool. Four is the decider's per-request image limit; with several cameras it takes fewer positions per camera.
- It sends one POST /v1/systemone request with three typed questions: quality as a choice over the three grades, worded to grade the execution and ignore the outcome; failure_mode as a choice over the eight tags plus none; and success as yes/no.
- It checks every answer's confidence against min_confidence (default 0.6). For a choice, confidence is the server's normalized confidence. For yes/no it is |2p - 1|, so 0.6 defers any probability between 0.2 and 0.8.
- If all three clear the bar, it writes the label through write_label, the same path every judge uses. If any answer falls short, it writes nothing and reports the episode as deferred, with all the probabilities.
The deferral covers the whole episode on purpose. A judge block needs a quality grade, and failure_mode set to none is a claim that nothing went wrong. Writing a half label would invent one of those, and the repo's rule is no silent defaults.
1
2
3
4
5
from strands_robots import DeciderJudge
judge = DeciderJudge("http://127.0.0.1:8765", min_confidence=0.6)
result = judge.judge_dataset(root, task="sweep the arm base past 0.3 rad")
deferred = result["content"][1]["json"]["deferred"] # episode indices for create_judge_agentThe backend keeps the repo's safety doctrine intact. A contrary success opinion becomes disputes_verdict: true, and the deterministic verdict stands. Existing labels, from a person, an agent or an earlier run, are skipped unless overwrite=True. Calls return the same status and content envelope as the other episode-judge tools and never raise mid-dataset: an unreachable server or one started without --vision is a single error up front, and a failed episode is reported while the run continues. The label records its provenance as strands-decider plus the served model name, so the revision travels with the data.
Building a test with known answers
The repo has no human-labelled holdout of recorded episodes, so I built one where the answers are known by construction. The task comes from the repo's example 19: sweep the base of a simulated SO-100 arm past 0.3 radians in MuJoCo, with a deterministic predicate that stops the episode on success. I drove the arm with the mock sinusoid policy and crossed two settings.
- Jitter: an alternating-sign offset of 0, 0.05 or 0.15 radians on every joint target, which maps to a high, medium or low quality grade.
- Budget: 15 steps, which runs out before the target, or 40, which reaches it.
At two amplitudes that gives 12 episodes, six successes and six failures. Quality and outcome vary independently, which is the same orthogonality test the repo applies to its VLM judge: a good judge should not grade a smooth failure as low quality just because it failed.
Alt text: Two rows of four camera frames from the MuJoCo SO-100 arm: episode 0, a failure with no jitter, and episode 11, a success with 0.15 rad jitter.
The four frames the decider sees for a failed episode (top) and a successful one (bottom).
The default scene camera shows the arm at about 40 pixels of a 640-pixel frame, which the decider's 448-pixel downscale makes almost invisible. So I also recorded the ladder from a camera framed on the arm, which is the run shown above.
Results
Alt text: Terminal output: the decider server and benchmark commands, a summary of 0 labeled and 12 deferred episodes, and episode 11's answers with confidences 0.18, 0.41 and 0.31 against a 0.6 threshold.
The benchmark run against v21 with the close camera. The per-episode block is formatted from the benchmark's JSON report.
| v19, default camera | v19, close camera | v21, close camera | |
|---|---|---|---|
| Labels written / deferred / failed | 0 / 12 / 0 | 0 / 12 / 0 | 0 / 12 / 0 |
| Forced success agreement (baseline 0.50) | 0.50 | 0.50 | 0.50 |
| Forced quality agreement (baseline 0.33) | 0.33 | 0.17 | 0.33 |
| Forced failure_mode agreement, n = 6 (baseline 0.33) | 0.33 | 0.33 | 0.33 |
| P(success) range | 0.29 to 0.31 | 0.35 to 0.39 | 0.31 to 0.39 |
| Success Brier score | 0.285 | 0.265 | 0.269 |
| Latency per episode, median (first) | 1.75 s (3.26 s) | 1.74 s (2.93 s) | 1.78 s (4.05 s) |
"Forced" means I took each answer's top option even when the backend deferred, to see what a backend without abstention would have written. The baseline is the score of always giving the most common answer.
Alt text: Bar charts: forced agreement for success, quality and failure_mode sits at the constant-answer baseline for all three runs; the backend wrote 0 labels and deferred 12 episodes.
Left: what the models would have written if forced. Right: what the backend actually did.
I read four things from this.
- Neither official model can separate these episodes from four still frames. Forced answers sit at the constant-answer baseline, and P(success) moves only between 0.29 and 0.39 whether the arm succeeded or not. v21 behaves the same as v19.
- Both models know they cannot. No yes/no probability left the 0.2 to 0.8 band, quality confidence never went above 0.21 and failure-mode confidence never above 0.43, all under the 0.6 bar, so the backend wrote nothing. A backend that always writes would have put 12 chance-level labels into the dataset.
- The quality answer did not collapse onto the outcome. The repo notes that an unsteered VLM judge tends to grade failures as low quality; here the grade was flat (mostly medium) across successes and failures. That is no signal rather than wrong signal.
- The cost per episode is small. About 1.8 seconds covers decoding the frames and one request answering three questions over four images on an Apple-silicon Mac, and the whole 12-episode run took about 23 seconds.
The ladder is hard for any image judge, and I picked it for that reason. Jitter is a tick-to-tick signal, and four evenly spaced frames cannot show it; the repo's own guidance is that smoothness lives in the state-jerk metric, not in frames. The success threshold is a few degrees of base rotation. A judge that is confident here without evidence is the failure I wanted to catch, and the backend caught it.
What I would do next
The obvious gap is a decider that has seen robot footage. Strands Decider already has a training recipe, and the simulator produces labels for free: the deterministic predicates give success, and settings like the jitter ladder give quality by construction. Fine-tuning a decider on episodes labelled that way, and scoring it with measure_agreement on a human-labelled real-robot holdout, is the follow-up I proposed in the issue. The same typed-decision pattern also fits elsewhere in the repo: an advisory task-completion check in the policy run loop, and a router that ranks registered policies for a scene while the agent stays in charge. Neither touches the human gate on motion, which stays the only approval.
Try it
- Issue: https://github.com/strands-labs/robots/issues/4572
- Draft pull request: https://github.com/strands-labs/robots/pull/4573 (53 unit tests against a fake loopback decider, a live MuJoCo test gated on STRANDS_DECIDER_URL, and the benchmark script in the description)
- Strands Decider: https://github.com/strands-labs/strands-decider
- Models: StrandsAgents/strands-decider-2B-hobson-v19 and StrandsAgents/strands-decider-2B-hobson-v21 on Hugging Face
The main thing I learned from this build: when a small model has to label data that trains a robot, a calibrated "I am not sure" is worth more than an answer. The backend writes labels the model can stand behind and hands over the rest, which lets a cheap model sit in front of an expensive judge without lowering the quality of the dataset.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article