Agents for Humans: The Answer Key Was Wrong — Auditing the Benchmark Before Blaming the Model

Part 6 of the Agents for Humans build series. Parts 1–3 covered the six-agent architecture, honest degradation, and a prompt quarantine layer; part 4 audited our own prototype and named its debts; part 5 paid the biggest one by shipping a ranking gate. This one pays the other debt part 5 left on the ledger — and reports what paying it uncovered: our benchmark's answer key was wrong, and seven of our twelve "misses" were never misses.

The Answer Key Was Wrong
We promised to measure it, and the measurement said no
Part 5 closed with two debts, stated in public. The first: Scout sampling variance — the same repository produced one hypothesis in one run and three in another, and we promised to measure that, not hand-wave it.

The measurement came back this month, and it is the least glamorous result of this entire series. We ran a threshold calibration ("tau study") over the gate's score distribution on four real open-source repositories — axios, pydantic, uv, pandas — looking for a cut line that would let the gate skip low-ranked hypotheses instead of merely reordering them. The raw model scores saturate: every raw score in the corpus — nine hypotheses across the four repositories and their gate re-run — is exactly 1.0. The demotions visible in the stored artifacts (three axios hypotheses and one pandas hypothesis, stored at −749) come from a rule-based degenerate penalty (−750), not from a score signal. There is no threshold to draw. Any cut line we picked would sit on top of a plateau.

So the honest conclusion is: the score channel cannot decide what to skip — only structural signals can. The gate v2 design ("from ordering to skipping") now has its scope narrowed to structural pruning before we write a line of it. That is what paying a debt looks like when the answer is no. The four repos' gate artifacts are committed (reports/v12-fp-gate-scores/), so you can look at the plateau yourself.

And then the study paid a dividend nobody priced in. Because before we could calibrate thresholds at all, we needed something we had been putting off: a ground-truth exam with an answer key we did not write ourselves.

The external exam humbles us
Our internal fine-tuning exam says the judge model (Laya) has a perfect record: FN = 0 across seven generations on an exam that is deliberately rebuilt to be meaner every round. We have written about how that number is earned — boundary pairs, paraphrased attacks, echo traps.

But an internal exam, however mean, is still self-graded. So we built the first external audit: 20 real web challenges from the NYU CTF Bench, run blind through the full pipeline, with every challenge's expected answer mapped by hand from official writeups. The scoring rule is one line: a hypothesis counts as a true positive only if it is CONFIRMED and its type matches the intended list, literally.

The first ledger, on the draft answer key:

Mechanical (literal match): TP 5 / FN 12 / indeterminate 3
Semantic (equivalence-class match): TP 7 / FN 8 / 5 / 1
For a team whose signature claim is FN = 0, missing twelve of twenty is the story. The normal script at this point is: tune the model, add recall, burn tokens. We did have that experiment designed and half-run — a "hybrid arm" that swaps the local Scout for a cloud model. Instead of finishing it, we paused at a cheaper question:

How many of those twelve misses are "the model didn't find it" — and how many are "our answer key was wrong"?

Auditing the answer key
The intended lists in our manifest came from reading other people's writeups. Writeups are notes, and notes get crossed, misremembered, and attributed to the wrong challenge. So we upgraded the method: for every failing challenge, we pulled the complete upstream source (partial clone, blob-by-blob — 51/51 files in the first batch) and required three independent confirmations before calling anything the "official solution": the challenge metadata, the official solve script, and the server source code itself.

Three flavors of wrong, from the first batch:

#9 whistleblow — our key said XSS / SQL injection / command injection, the standard web trio. The directory contains no web application at all. It is a pure AWS scavenger hunt: dig four pieces of metadata out of 200 pseudo-random objects in one S3 bucket, use them to reach a second bucket. The flag self-reports: flag{pwn3d_th3_buck3ts}.
#10 United — our key said SSTI + RCE. The official README's overview spells out the actual chain: SQLi Union attack to leak admin hashes, crack the SHA-1, find the unlinked admin page. The vulnerable line is template-string concatenation in routes/players.js:12, and the official sample payload matches it verbatim. The flag: flag{United_w3_leek_the_d4atabas3}.
#18 gatekeeping — our key listed a JWT vulnerability. There is zero JWT or token logic anywhere in the tree. The real chain abuses gunicorn's habit of loading WSGI SCRIPT_NAME from a request header, smuggling /admin/key past an nginx prefix check. The flag: flag{gunicorn_probably_should_not_do_that}.
And in all three, both generations of our Scout had, in fact, produced findings matching the real solution. The pipeline was right more often than our answer key.

Four batches later: the ledger moves without burning a token
We audited all twelve failing challenges this way, in four batches, fixing the answer key only where source evidence demanded it. Then we re-scored — same judgments, new key, zero model tokens:

Seven misses flipped to TP on the mechanical ledger (United, gatekeeping, nft-world, nekocat, unagi, poem-collection, real-time-chat), and an eighth (whistleblow) flipped on the semantic ledger.
Final state: mechanical TP 12/20, semantic TP 13/20.
Along the way we fixed an accounting bug we had shipped in our own scoring narrative — the semantic ledger once summed to 21 out of 20 because one challenge was double-counted. The fix is in the table; the embarrassment stays in the table too.
One flip deserves its own paragraph because of how cheap it was. The real-time-chat challenge's key finding was titled, in natural language, "Remote Code Execution / Privilege Escalation" — a composite string that never literally matched our REMOTE_CODE_EXECUTION token, so the pipeline's correct call sat in the miss column across every ledger version. The fix was one normalization in the scorer (split on slashes, case-fold each segment — substring matching deliberately rejected as too loose), plus three guard tests including the exact historical string. Recomputed ledger: 19 of 20 questions moved by exactly zero, and the wrongful conviction turned into a TP. Every scorer version (v01 through v05) is committed; none of the published numbers were rewritten retroactively.

What a bigger Scout did and did not fix
Now the other half of the debt: the hybrid-arm experiment, half-run before the audit intervened. Only the Scout stage changes — a stronger cloud model — while verify/deepen/arbiter stay on our local four-service stack. Same questions, same judge, one variable.

The good news first: on the first two re-run challenges (whistleblow, gatekeeping), the hybrid arm flipped both to TP, and its findings reached deeper than the local Scout's — source-level hits where the local generation had stopped at documentation level.

Then the cold water. On real-time-chat, the cloud Scout recalled three times the hypotheses (4 → 12) — configuration, DoS, supply chain, key injection, data exposure, the full periphery. Of those twelve, three confirmed, and all three are real weaknesses: unauthenticated writes to a shared list, unbounded Redis keys, unsanitized session IDs as key names. But the official chain — TURN proxy abuse reaching an internal Redis, then replica-of replication plus module loading for RCE — never surfaced. Breadth tripled; the chain stayed invisible.

Here is the part we are most glad we caught before publishing: our own conclusion from that run was wrong too. The internal note said the hybrid arm "hit the official XSS delivery primitive but showed a blind spot on sink reasoning." The audit overturned it — that hypothesis pointed at the frontend rendering surface, a real weakness but not the official solution, and the official solution had actually been found by the cheaper local model, then buried under our wrong answer key. Half of what we were about to call a "model blind spot" was another wrongful conviction. The lesson generalizes: a "blind spot" conclusion is itself a claim, and it goes through the same source-level audit as everything else.

After four batches, the twelve original misses separate into two clean buckets:

Seven wrongful convictions — the answer key's fault; fixed by auditing the key and one line of scorer normalization, zero model tokens. (One more challenge, search_for_pi, turned out to be a pure logic-puzzle — out of scope for a vulnerability pipeline, recorded as "type mismatch," not a detection failure.)
Six real misses — the key was right, and no generation of Scout, local or cloud, ever produced the official chain: a logging-config fileConfig RCE behind a missing admin check; a cache-key overwrite feeding pickle; cURL globbing bypassing a URL validator into a Caddy admin API; an EPS file riding a Ghostscript parser; a Perl diamond-operator command pipe; a FAT-filesystem import gadget. What they share is that the pivotal step lives outside the prior distribution of "typical web vulnerabilities." No scout we can rent fixes a prior that isn't there.
So the next experiment — queued, not yet run — changes shape: feed the four hardest of those six to the hybrid arm and record, per challenge, whether the cloud Scout can raise chains like fileConfig or cache-key poisoning at all. If it cannot, the fix is priors and rules, not a bigger model.

The receipts
Everything above traces to committed files: the audit document (docs/K2-INTENDED-REVIEW.md, every claim pinned to flags, payloads, and line numbers), five coexisting score files (v01, k2, v02, v03, v04, and since this week v05 — the draft-era numbers are still in there, because correcting history honestly means keeping it), the scorer and its tests (eval/m8_k1_score.py, including the normalization guards), the four repos' gate artifacts (reports/v12-fp-gate-scores/) plus the pandas gate re-run (reports/v12-fp/pandas-e2e-gate/). The internal-exam numbers live in docs/EVAL-LEDGER.md.

Update (Oct 16): round two started paying, and the miss list moved again
The buckets above were frozen on Oct 13. Round two of the hybrid arm has since run its first clean-topology passes, and the six-real-misses bucket already shrank — in a way that matters more than the shrink itself.

good_intentions came off the list entirely. The profiled cloud Scout raised the fileConfig chain as its top hypothesis, verify confirmed it, and the finding's type matched the intended list literally — a composite REMOTE_CODE_EXECUTION / PATH_TRAVERSAL label that the scorer's segment normalization catches. Miss → TP, zero retries.
flask_caching turned out to be a different kind of miss. Our local Scout had raised the official chain all along — its top hypothesis was labeled DESERIALIZATION / CACHE_POISONING, the cache-key-overwrite-feeding-pickle chain word for word — but verify returned UNCERTAIN on every hypothesis, so nothing entered the ledger. The recall was there; the verdict was not.
webhooktester split the difference. SSRF — the intended class — was raised as H1 and also came back UNCERTAIN, while the two hypotheses verify did confirm carried labels outside the intended list.
So the honest count is now three real misses (the Ghostscript EPS parser, the Perl diamond operator, the FAT import gadget — none re-run yet) plus two reclassified cases: the scout did its job and the judge flinched. The claim "no scout we can rent fixes a prior that isn't there" survives for exactly those three; for the other two, the next experiment is not a bigger Scout but a verify stage allowed to commit — which is queued as its own arm.

One more scorer note, since it belongs in the same audit trail: v05 added a code-execution equivalence pair and was re-scored against all twenty questions with zero movement — the ledger above stands.

What we still owe
Two new debts, in public, same deal as always:

Hybrid arm, round two — the blind-spot experiment above, scored per challenge against the real misses. It will tell us whether "rent a stronger scout" has any remaining headroom on chains the prior doesn't cover.
raid_fat_fs_chal's bridge — the upstream repo ships no official solution; our key rewrite is source-evidenced (an unfiltered dynamic import endpoint, a FAT-mounted webroot), but the full upload-to-import path stays honestly un-reconstructed until better material surfaces.
Part 4 ended with a debt list. Part 5 paid one. Part 6 pays another and finds the answer key sitting on the debtor's side of the table — which, honestly, is where we should have looked first, because it was the one component in the whole system that nobody had put under audit.

The repository: a9320/arcanum-spark  (MIT). If you scan your own repository and disagree with a verdict — or with our answer key — that is the most valuable issue you can file, and the template is waiting.