How I Trained the Best 2B Strands Decider: Beating the Official Strands Model and the 2B Leaders, Top 3 in Text, Image and Multimodal

I trained the best 2B Strands Decider models to date. Four new checkpoints beat the official Strands model, the published Strands Decider v19, on every headline measure I ran. By my estimate they take the top 3 spots on the 2B-class JevBench text board, place #2 on Image JevBench, and give the strongest single 2B checkpoint for text and images together. On 1,050 questions from task families that no model trained on, every model of mine that I measured outscores the two current official 2B leaders, decider-2b and decision-2b, which I ran locally with their own published code.

Alt text: Results overview: estimated #1 in text (53.6 v1.5-style Intelligence against 29.5 for the official Strands Decider v19), #2 in image (43 of 60 Image JevBench preview items against 40) and #1 multimodal with one model for text and images; +82 percent Intelligence over v19 and an estimated +4.5 Capability over the current 2B #1; a text leaderboard where minicpm-v21-vl (~63), hobson-v20 (~61) and minicpm-v21 (~61) sit above decision-2b (58.8), decider-2b (56.9) and Strands Decider v19 (~55). Marked unofficial, submission pending.

The work happened in two steps. First I unlocked vision: my PR #10  , merged upstream on 3 October 2026, gave Strands Decider image input through a --vision flag without changing any weights. Then, over 3 and 4 October, I set out to train the best 2B model on top of it.

The starting problem was decisiveness. v19 looked like #3 of 33 on the old accuracy board, but under the current v1.5 rules a yes/no answer with a probability between 0.2 and 0.8 counts as wrong, and v19 lands in that band on 48 of its 74 yes/no tasks. The gains came from four changes: a 27B teacher for the yes/no answers, with the anchor that held them close to the frozen base model removed; a base-model bake-off that picked MiniCPM5-2B under a rule written before any score; a temperature refit for choice and score questions that I call Fix A; and an image stage that keeps the text gains.

The rest of this article follows the work in the order it happened: how the benchmark scores a decision model, the reproduction of v19, each training idea with its measured result, the runs that failed and why, the unseen-family check, and how the four candidates were packaged. The failures get the same space as the wins.

Everything here is exploratory, local and unofficial; no number is an official JevBench or Image JevBench score. Every board position below is my own estimate, given with pessimistic, middle and optimistic bands, and the official submission is still pending.

Alt text: Lineage diagram that starts at the published v19 checkpoint and branches into the --vision image input, the v20 image fine-tune, v20 short, v20-long, the failed first combined model and its fix, Fix A, the base bake-off, v21 on MiniCPM5, the SigLIP2 graft and the rejected Gemma 4 E2B line, ending in four packaged candidates

Strands Decider (strands-labs/strands-decider  ) is a small decision model from Strands Labs. It answers yes/no questions (the benchmark calls the type "noul"), multiple-choice questions and score questions, where the answer is a level on a scale. It returns a probability distribution over the options, so a grader can check both whether the answer is right and how confident the model was. A base language model (the torso) carries a LoRA adapter, and a small pointer head reads out the option numbers. v19 is built on Qwen3.5-2B-Base.

JevBench v1 public has 231 tasks: 74 yes/no, 139 choice and 18 score. Three rules of the v1.5 scoring drive everything below.

The first is the band rule. A yes/no answer with P(yes) strictly between 0.2 and 0.8 counts as wrong, whichever side of 0.5 it lands on. Yes/no competence is then 100 * (acc - 0.5) / 0.5, so a model that hedges on many questions can score below zero on the type. Choice is chance-corrected as 100 * (acc - c) / (1 - c). Score questions are graded by the expected level of the returned distribution, as 100 * (1 - nMAE / nMAE_chance). Intelligence is the mean of the three types.

The second rule defines the headline number: Capability = (Intelligence + Calibration) / 2, so calibration carries half the weight. Working through the formula, yes/no calibration comes to 100 * (1 - 2 * ECE), where ECE is expected calibration error, and the Brier score plays no part in it. The ECE I report throughout is top-label ECE over 10 bins.

The third rule is the sealed half. Official Intelligence is 50 percent open tasks and 50 percent sealed tasks that only the maintainers can see. A local run measures the open half and nothing else.

The band rule has a side effect that shaped several decisions. Sharpening yes/no probabilities past 0.8 or below 0.2 turns band answers into scored answers, and the composite rewards that even when the sharpened probabilities are miscalibrated. In the scoring-rule sense the composite is improper. The expected-level grader is improper as well, because the median of a distribution is not its mean.

I reimplemented the v1.5 rules locally and call the result "v1.5-style Intelligence". Early on I used an older local script that graded score questions by the top level instead of the expected level. It overstated some models, so every number from it is labelled "proxy".

To tie every later gain to a training change, I first reproduced v19's published score. I rebuilt the corpus, checked each file against the repository's data/SHA256SUMS, and re-ran JevBench v1 at a 4096-token window on a single rented H100. The run took about 13 minutes.

The result was 167 of 231 tasks right, identical task for task, with a maximum probability difference of 0.0. train_v5.jsonl (100,449 rows), multistep_v14.jsonl (12,909 rows) and every committed synthetic file matched their hashes. holdout_v5_norule.jsonl (21,000 rows) did not, because an upstream held-out Hugging Face dataset had changed, so all of my calibration ran on a rebuilt copy of that file. That rebuilt file later became the most likely cause of a score-temperature bug, described in the Fix A section.

On the retired v1.4.2 board, which ranked by public accuracy, v19 scored 0.723 (167 of 231) and placed 3rd of 33 in the 2B class. It had never been measured under the v1.5.5 rules.

On my v1.5 reimplementation v19 scores 29.5 Intelligence (the old proxy said 32.1). Its proxy competence by type was 65.4 on choice, 63.4 on score and -32.4 on yes/no. The cause of the yes/no number is in the hedging count: 48 of the 74 yes/no answers sat inside the 0.2 to 0.8 band, and 26 of those 48 leaned toward the correct answer. ECE was 0.050, so v19 was well calibrated and also unwilling to commit, and v1.5 scores that unwillingness as failure.

The v19 recipe trains with a frozen-KL anchor: an extra term, KL(frozen || student) over the option-number readout at weight 0.3, that keeps the student close to the untouched base model. On a Base model that anchor sits near chance on yes/no, so on those rows it pulls the student toward a 50/50 answer, which is exactly a band answer. My hypothesis was that skipping the anchor on yes/no rows and distilling a more decisive teacher on those rows would shrink the band without hurting the other types.

Skipping the anchor is one line of config, kl_frozen_skip_kinds: ["noul"], which zeroes the KL term for yes/no rows and leaves it at 0.3 everywhere else. The frozen readout is precomputed once per run, which took 899 seconds on a 2B full epoch.

Before trusting a teacher I checked it. Qwen3.5-27B scored 0.744 on a 234-row HelpSteer2 adequacy eval, against 0.645 for a frozen 4B model and 0.739 for v19. On a 302-row generated adequacy eval it scored 0.884, against 0.810 and 0.788. I had the 27B label about 55K yes/no rows, kept the 45,337 where its argmax agreed with gold, and merged them with 12,909 replay rows into a 58,246-row teacher file. One detail matters for anyone reproducing this: the teacher file indexes rows by their position in the concatenated train_files, so the order of training files must not change between labelling and training.

I ran four arms, each continuing v19 for 1,500 steps on a rented 4x H100 machine: more training with no other change, no yes/no anchor, the 27B teacher, and the teacher plus no anchor.

More training alone helped choice tasks and barely moved yes/no (band 44). The teacher arm looked strong on seed 0, so I ran seeds 1 and 2 on both promising arms. The teacher alone scored 39.4, 31.8 and 32.1, and across those seeds its paired gain over v19 was +2.3 with a 95 percent interval of -1.7 to +6.6, inside noise. The combined arm held at 40.6, 37.4 and 42.3. Against v19 its paired intervals were -11.3 band answers (-17.7 to -5.3) and +8.0 proxy Intelligence (+2.9 to +13.3), while tasks right moved by +3.3 (-3.3 to +10.3), within noise. In v1.5-style units this "v20 short" scores 37.7.

I kept teacher plus no anchor as the text recipe. The gain was real and modest, and about half of the yes/no answers were still in the band.

In parallel I worked on images, building on the --vision input from PR #10. I measured images on three public sets. NaturalBench (NB) has 300 groups of 1,200 questions in total, and its group accuracy (G-Acc) credits a group only when every question in it is right. POPE-adversarial (600 questions) tests whether the model claims objects that are not in the picture. The third set is 60 Image JevBench preview items that I could rebuild locally. I also tracked blind confidence: the model's mean confidence when the image is removed from the question.

The first experiment changed no weights. At a 448-pixel long-side setting the preview score was 37 of 60. A 400k-pixel budget raised it to 40, with NB 0.788, while POPE fell from 0.877 to 0.858. Per-kind image temperatures, fitted by NLL on NB groups 300 to 599 over a 601-point log grid in [0.3, 6], changed only calibration (Image JevBench ECE 0.089 -> 0.072). I kept the 400k budget and the temperatures.

The second experiment was a fine-tune on about 38K image rows (VQAv2 pairs, COCO counting, Mind2Web, TabFact and my own chart and document renders) plus 3,000 text-replay rows. I deduplicated 21,704 training images against 1,828 evaluation images by perceptual hash and found nothing within Hamming distance 6. COCO overlap with POPE was zero, and I dropped one Mind2Web website after finding six near-duplicates of preview items.

v19 answered confidently even when the image was missing, so variant A added image-removed copies of about 15 percent of the image rows. Those copies carry no label weight and are trained only by KL toward the frozen torso's text-only reading. Variant B had no such copies. Over three seeds A scored NB 0.805, POPE 0.874 and 42.7 of 60 on the preview, and B scored 0.802, 0.881 and 43.0, a tie on accuracy. With the image removed, A's confidence fell to 0.58 on NB and 0.53 on POPE. B barely moved on NB (0.615 against v19's 0.652) and stayed high on POPE (0.664 against 0.725).

I took A forward because it fixes the blind-confidence failure, and I recorded that it missed one of my own preregistered bars: group accuracy came in at 0.371 against a bar of 0.383, and NB blind confidence sat at 0.582 against a line of 0.58. These text and image changes are proposed upstream as code only, all opt-in, in PR #23  (open), together with everything from the later sections; it passes 437 tests (22 skipped) with clean ruff and mypy.

Next I tested whether another 2B base would learn the v20 recipe better. At 20:10 UTC on 3 October, before any arm had produced a score, I committed the decision rule. A non-Qwen base would win only if its proxy Intelligence beat the higher of two Qwen seeds, which served as the noise floor. If two non-Qwen bases cleared that bar within 1.0 point of each other, lower Brier score would decide, then lower median latency.

Each arm trained from the raw base for 2,400 steps (about 0.64 of an epoch) with the same data and recipe on a rented 4x H200 machine. The LoRA settings follow v19:

In proxy units MiniCPM5-2B scored 48.5 with 179 tasks right, the two Qwen seeds scored 30.0 and 40.8, and Gemma 4 E2B scored 14.5 with 150 right. MiniCPM5 beat the Qwen mean by +13.0 (+6.3 to +20.0), and Gemma trailed the Qwen mean by 20.9 (-32.0 to -9.6). The two Qwen seeds differed by more than ten points, which showed why using the higher seed as the noise floor was a sensible rule.

MiniCPM5 won under the rule I had fixed in advance. MiniCPM5-2B has 2.5B parameters, so its entries sit in the 2B class with an asterisk, the same one decision-2b carries. For that reason I kept a strictly-2B Qwen line running alongside it.

Two full-epoch runs came next on the v20 text recipe. An epoch is 119,623 rows / 8 / 4 = 3,738 steps. Each run had three seeds, and I wanted to average their weights afterwards into a single checkpoint, which I call the soup, so I split the seed in the code: init_seed controls initialisation and seed controls the validation split, data order, option shuffles and dropout. All three runs share one initialisation.

The soup is exact for LoRA. Each run i has an update s * B_i A_i with rank 16 and scaling s = alpha / r = 2. The mean of three updates is (1/3) * sum_i 2 * B_i A_i, which equals (2/3) * [B_1 B_2 B_3][A_1; A_2; A_3]. So I concatenate the factors to rank 48 and set the scaling to 0.666667, which reproduces the mean update with no approximation. The head is averaged element-wise.

v21 trained MiniCPM5-2B from raw. As trained it scored 38.9 v1.5-style (39.9 / 39.7 / 37.2), with 179.3 tasks right, the best count of the project, and 27.0 hedged. Against v19 it gained +18.3 proxy (+10.6 to +26.7), but against the bake-off MiniCPM5 its gain of +2.0 (-1.8 to +5.9) was within noise, and its soup scored 37.8, no better than its seeds. Its ECE of 0.088 was the warning sign.

v20-long continued from v19. Its seeds scored 30.1, 37.1 and 39.8 v1.5-style, and the soup scored 39.7 with 177 right and 35 hedged. In proxy units the soup reached 41.5 against a seed mean of 38.7, a +2.8 gain from averaging.

I compared per-kind temperatures (yes/no / choice / score) across checkpoints on 3,833 held-out rows. v19 had 0.91 / 0.73 / 1.33. The three v21 seeds and the bake-off MiniCPM5 had score temperatures between 3.69 and 4.58, with the v21 soup at 2.16. Dividing logits by 4 flattens a score distribution, so the expected level drifts toward the middle of the scale and the expected-level grader punishes it. The proxy graded by the top level, which a flat distribution does not move, so it hid the damage. The most likely cause was the rebuilt holdout_v5_norule.jsonl, the one file whose hash had not matched. Separately, every MiniCPM5 checkpoint was over-confident on yes/no (+0.051 to +0.094, against +0.014 for v19), the price of the decisiveness the 27B teacher taught.

Fix A refits the choice and score temperatures by negative log-likelihood on held-out rows, fitting on one half and checking on the other. It leaves the yes/no temperature alone, because any change to the yes/no temperature moves answers across the band edges. An honest NLL refit of all three kinds on v21 seed 0 showed the effect: hedged answers rose from 21 to 39 and proxy Intelligence fell from 51.5 to 43.3.

Fix A improved every MiniCPM5 checkpoint on every check. Public Intelligence rose by about 4 to 7 points (5 to 7 on the v21 and bake-off checkpoints, 3.6 on the graft), public ECE fell by a third to a half, held-out score ECE roughly halved, ECE on unseen families fell, and score-type Intelligence rose from about 36 to 56-58. v21 went from 38.9 to 45.8 (47.1 / 46.2 / 44.2), and the v20-long soup went from 39.7 to 42.5.

Alt text: Three dumbbell panels showing values before and after Fix A. Public Intelligence rises for v19 (29.5 to 34.2), the v20-long soup (39.7 to 42.5), the bake-off MiniCPM5, the v21 soup, the v21 three-run mean (38.9 to 45.8) and the MiniCPM5 graft (47.7 to 51.3). Public ECE falls for three MiniCPM5 checkpoints, and unseen-family ECE falls for v19, the v20-long soup and every MiniCPM5 checkpoint

The rejected idea came out of the same session. A "balanced" refit chose the yes/no temperature to maximise (competence + calibration) / 2 directly. On v21 seed 0 it moved proxy Intelligence from 51.5 to 63.2 and hedged answers from 21 to 2. On v19, with no training at all, it moved 32.1 to 52.0. The chosen temperature was 0.22, the floor of my search grid. A temperature that low pushes every yes/no probability out of the band whatever the model believes, which is clamping, and "no probability clamping" was already one of my written rules. Because the composite is improper, those points say nothing about what the model knows. I rejected the refit as metric gaming and kept its output under a name that says so: calib_balanced_REJECTED.json.

To get a single 2B checkpoint for both boards, I ran the variant-A image stage on top of the v20-long soup. Images came out fine: NB 0.801, POPE 0.877, preview 40.7, NB ECE 0.023. Text fell from 39.7 to 29.3 v1.5-style, below v19.

The cause was in the image stage's configuration. It re-applied the frozen-KL anchor to every row, including the yes/no text-replay rows, and it replayed text with plain gold labels instead of teacher targets. In effect the image stage trained the hedging back in: the first combined model put 47.0 yes/no answers in the band, against 35 for the soup it started from.

The fix applied both v20 ideas inside the image stage. The repaired run (I call it combfix) started from the v20-long soup with text_replay_n 3000, ablation_fraction 0.15, kl_frozen_weight 0.3, kl_only_weight 1.0, kl_frozen_skip_kinds: [noul], image_max_pixels 400000, lr 5e-5, head_lr 2e-4 and 32 rows per step for one epoch, with 27B-teacher targets on the replay rows. It scored 39.0 on text (39.4 / 39.0 / 38.7) with images essentially unchanged: NB 0.802, POPE 0.874, preview 42.0, blind confidence 0.60 and 0.58. Its seed standard deviation on text was 0.35, the tightest of any configuration I trained.

Seed 0 became the package hobson-v20-balanced. Its image temperatures were fitted (1.93 for yes/no, 1.07 for choice) but are not applied in the package, and it was not run on the unseen-family set.

Gemma 4 E2B finished last in the bake-off, so I ran a short recipe sweep before giving up on it. The instruction-tuned E2B-it did best, with 177 tasks right, on par with hobson-v20 and close to the best, yet 57 of its 74 yes/no answers sat in the band, and it scored 17.4 v1.5-style. My log entry reads "Gemma knows the answers but hedges." With the image stage added, Gemma fell to 10.0 on text and reached NB 0.729, POPE 0.836 and 27 of 60 on the preview, below v19 with --vision. I rejected it.

MiniCPM5-2B has no vision tower, so for the MiniCPM5 line I grafted one on. I attached SigLIP2-so400m (patch 16, 384 px), used a 2x2 unshuffle to get 144 tokens per image, and aligned an MLP projector (4608 -> 2048 -> 2048) on 80,000 COCO captions in 694 seconds. Then I ran the variant-A image stage from the v21 soup.

On text the grafted model gave the best numbers of the project: 51.3 with Fix A (49.8 / 53.6 / 50.4) and only 16.3 hedged. The images were poor: NB 0.679 against v19's 0.787, group accuracy 0.130, POPE 0.818 and 16 of 60 on the preview, below Gemma on every image accuracy metric. Seed 1 became the text-first package minicpm-v21-vl, and the lesson I recorded is that a graft needs a far larger alignment stage than 80K captions. For a model that has to read images well, the Qwen line, which kept v19's own vision path, stayed well ahead.

Alt text: Four bar panels comparing NaturalBench accuracy, NaturalBench group accuracy, POPE accuracy and the 60-item preview score for v19 with --vision (0.787, 0.347, 0.858, 40), hobson-v20-balanced seed 0 (0.802, 0.360, 0.865, 43), minicpm-v21-vl seed 1 (0.675, 0.130, 0.817, 16) and Mapika decider-2b-vision as a reference (0.794, 0.360, 0.860, 38)

Alt text: Bar chart of v1.5-style text Intelligence on the 231 public tasks, with dots for seeds 0, 1 and 2 where they exist and a dashed line at v19's 29.5: v19 29.5, v20 short 37.7, hobson-v20 42.5, combfix 39.0, v21 with Fix A 45.8, the MiniCPM5 graft with Fix A 51.3, and Gemma text plus images lowest at 10.0

By this point MiniCPM5 led every public text table, but half of official Intelligence comes from sealed tasks. To see performance on tasks the models had never seen in any form, I built 1,050 rows from five families that appear in no training mix, checked against the repository's data sources list: StrategyQA and RuleTaker depth-5 for yes/no, CommonsenseQA and ARC-Challenge for choice, and STS-B at six levels for score.

I also ran the two strongest 2B entries locally with their own published inference code and calibration: Mapika's decider-2b, the 2B leader on official Intelligence (42.3), and FlyMy's decision-2b, the 2B leader on official Capability (58.8). Their order on my set (21.0 above 20.2) matches their official Intelligence order, which is the one sanity check I have that the set tracks something related to the sealed half.

Every model I trained and ran on this set scored above both competitors. The lowest, v21 seed 2 at 26.6, is 5.6 points above decider-2b, and hobson-v20 is 13.7 above it. The lead comes from yes/no. On choice and score decider-2b is better: 73.8 and 65.7, against 64.7 and 50.1 for hobson-v20 (the best score result in my table is v19's 51.1). decision-2b beats hobson-v20 only on score (57.7 against 50.1).

The order among my own models changed. The v20-long soup dropped 7.8 points from public to unseen and had the lowest unseen ECE of the trained models at 0.055. The MiniCPM5 models dropped 17 to 21 points. MiniCPM5's public lead is largely familiarity with the public task formats.

Alt text: Dot plot pairing public and unseen-family Intelligence for each model after Fix A. The MiniCPM5 graft seeds and v21 checkpoints drop 17 to 21 points, v19 drops 10.6, the v20-long soup drops 7.8 to an unseen score of 34.7, and decider-2b (21.0) and decision-2b (20.2) appear with unseen scores only

Two cautions apply. Every model I ran, the competitors included, is below chance and over-confident on unseen yes/no, so none of them handles that part well. And this is a single set of 1,050 rows, so I treat differences under about 3 points as noise.

I packaged four checkpoints, and each package is one specific seed, because a mean and a chosen seed are different claims. hobson-v20 is the v20-long soup with Fix A. hobson-v20-balanced is combfix seed 0 (NB 0.802, POPE 0.865, preview 43 of 60). minicpm-v21 is v21 seed 0 with Fix A. minicpm-v21-vl is the graft's seed 1 with Fix A, which was its best seed, chosen after I saw the results.

To place them on the board I had to estimate two things I cannot measure. For the sealed half I used my unseen score plus an offset of 7.8 to 14.9, measured on the two competitors as their official sealed score minus my unseen score for them: 35.9 - 21.0 = 14.9 for decider-2b and 28.0 - 20.2 = 7.8 for decision-2b. Calibration cannot be measured locally, so I used a band from 72 to 85. Capability is then (0.5 * open + 0.5 * sealed + Calibration) / 2, evaluated at the pessimistic, middle and optimistic ends. With two anchors behind the offset, these are rough estimates.

* hobson-v20-balanced was never run on the unseen-family set, so it has no sealed-half estimate and no Capability band. Its claims rest on public text (39.4) and the image sets. The v19 row uses its uncalibrated scores from before Fix A (34.2 public and 23.6 unseen after it).

Every rank in that table is a local, unofficial estimate. At the middle estimate minicpm-v21-vl would place first in the 2B class, hobson-v20 first among strictly-2B entries and minicpm-v21 third, which is where the top 3 claim comes from. At the pessimistic end every candidate except minicpm-v21-vl falls below decision-2b's official 58.8. All the bands overlap the official leaders, so these positions hold only until an official score replaces them. On Image JevBench, preview items put hobson-v20-balanced at about #2 in the 2B class, behind imajev 2B (74.1) and ahead of Mapika (67.1), which is also a local, unofficial estimate. An all-sizes rank cannot be estimated because of the Speed and Cost axes and the eligibility gates.

Alt text: Horizontal range bars of estimated 2B-class Capability for minicpm-v21-vl (59.4 to 67.7, middle 63.3), hobson-v20 (57.2 to 65.5, middle 61.1), minicpm-v21 (56.9 to 65.2, middle 60.8) and v19 (50.4 to 58.7, middle 54.6), next to vertical lines for the official scores of decision-2b at 58.8 and decider-2b at 56.9

The pick depends on the goal. hobson-v20 is the strictly-2B text entry that generalises best. hobson-v20-balanced is the one 2B checkpoint for both boards, slightly ahead of v19 on images and lifting text from 29.5 to 39.4. minicpm-v21-vl has the highest public text score, but it is 2.5B plus a 0.4B vision tower, drops the most on unseen families and has weak images. minicpm-v21 gets the most text tasks right: 178 on its packaged seed and 182 on its best.

Each archive holds the LoRA adapter, the head, calibrated temperatures, a model card, SHA-256 sums and the exact training config, and the bases download from Hugging Face at pinned revisions. Serving all four packages needs the opt-in code in PR #23, which includes the MiniCPM5 multi-base support and the graft code. The weights are public, with checksums, and I have offered them to the maintainers in issue #24  . The maintainers will decide which, if any, to submit under the Strands name.

All training ran on GPUs I rented on vast.ai. The v19 reproduction and a pixel-budget study each ran on a single H100, the v20 text arms and the image fine-tune shared a 4x H100 machine, and everything from the bake-off on ran on two 4x H200 machines.

Amazon S3 was the artifact store. Data, checkpoints, soups and temperature files went to a private bucket, and the rented hosts read and wrote through short-lived presigned URLs, so no AWS credentials ever sat on a machine I did not own. Each machine was destroyed only after its uploads were verified.

The hardware was underused. A 2B full epoch took about 2.1 hours end to end, peak memory was 12.4 GiB on a 141 GiB H200, and model FLOPs utilisation (MFU) was about 10 to 15 percent. No H200 run had causal_conv1d installed, so every run used the slower reference kernel. Installing the fast kernels would have been an easy speed-up.

Decision rules were written down before the scores they judged. The v20 bars were preregistered for the mean of three seeds with no seed worse than v19: proxy at least 37.0, band at most 40 of 74, text ECE at most 0.07, NB at least 0.800, POPE at least 0.865 and blind confidence at most 0.60 on both sets. When variant A missed the group-accuracy bar, the miss went into the PR text. The bake-off rule was committed before any score existed, and the image dedupe threshold was fixed before training.

Every data file and package has a SHA-256 hash, and the two hash failures are logged as failures: the held-out file from the reproduction, and VQAv2 and Mind2Web image files that differed in bytes between two builds with identical row counts. Base models and the teacher are pinned to exact Hugging Face revisions, and both 27B teacher files are identified by hash. Rejected results keep their files under names that say they were rejected. No benchmark items or paraphrases went into training.

Splitting init_seed from the data seed is what makes the soups valid, and seed spread varied a lot by recipe, with standard deviations of 2.6 for v20 short, 5.0 for v20-long, about 7.9 for Qwen from raw, 1.5 for v21 and 0.35 for combfix. Not every configuration ran three seeds: the bake-off used seed 0 (two seeds for Qwen), the v20 control and no-anchor arms ran seed 0 only, and the Gemma diagnostic arms were single runs. The unseen-family and Fix A deltas in this article carry no intervals.

These runs are exploratory and local, and I chose later arms after seeing earlier results. The text scorer is my reimplementation of the v1.5 rules, and both the sealed half and Calibration are estimated, with the sealed offset resting on two anchors. The grafted model's image numbers come from public sets, and the MiniCPM5 checkpoints are 2.5B. The bake-off ran one seed per base (two for Qwen), and the fixed combined model is one configuration with three seeds; differences under about 3 points are noise. The packaged candidates are single seeds, the minicpm-v21-vl seed was chosen after results, and hobson-v20-balanced was not run on the unseen-family set.

There is also a naming caution. The Strands team ran its own internal experiment called v20, a different model that failed four of six predictions, after which v19 stayed the reference. My v20 is unrelated, which is why the packages are named hobson-v20.

The code is in PR #23  on strands-labs/strands-decider  , stacked on the merged image-input PR #10  . The experiment branches and the results log live on my fork at Vivek0712/strands-decider  . The four candidate packages are offered to the maintainers in issue #24  , where they will decide what gets submitted; no official submission has been made yet.