Agents for Humans: The Gate That Spends Wisely — Paying the Ranking-Gate Debt
Part 4 named Debt 2: our pipeline had no ranking gate. Part 5 pays it — a fine-tuned ModernBERT judge that reorders our six-agent verification budget (AP=1.0 at the peak, zero false negatives across every shipped generation), the production wiring gap we caught and fixed this week, and a four-repository false-positive campaign that ended at zero, with every raw artifact committed for review.
Part 5 of the Agents for Humans build series. Parts 1–3 covered the six-agent architecture, honest degradation, and a prompt quarantine layer; part 4 audited our own prototype and named its debts. This one pays Debt 2: there was no ranking gate.
Agents for Humans, Part 5 — The Gate That Spends Wisely
Why the budget, not the model
When we audited our prototype in part 4, the debt list had a clear headline item: our six-agent pipeline verified hypotheses in the order the Scout happened to emit them. That was fine when Scout proposed four hypotheses. It stopped being fine the day it proposed twelve.
Verification is not free. Every hypothesis that reaches the Verify stage costs tokens, latency, and — worst of all — reviewer attention when the report lands. A pipeline that verifies everything in emission order has no opinion about what matters. The fix is not a bigger model. The fix is an opinion.
So we built one: Laya, a fine-tuned ranking gate that reorders hypotheses by expected verification value before a single token of Verify budget is spent.
Meet Laya: eight generations of a tiny judge
Laya is a ModernBERT-base checkpoint fine-tuned on our own exam corpus — KEEP/PRUNE decisions over hypothesis-shaped inputs, ranked by probability. No agent framework, no tool calls, no chain of thought. It reads a hypothesis and its scout-stage context, and it outputs one number.
The lineage tells the story better than the architecture diagram:
- r3 scored 20/24 on our held-out exam (AP 0.958) — and, crucially, zero false negatives: every real threat in the exam survived.
- r4 climbed to 21/24. r5 hit 23/24 and produced calibration good enough to ship (Brier 0.033).
- r6 went 24/24 — a perfect ranking, AP 1.0.
- r7 traded one accuracy point for honesty about uncertainty: ECE dropped from 0.1016 to 0.0417. We kept r7 in the seat; a gate that knows when it is unsure is worth more than one more correct answer.
- r8 tried to fix the last stubborn wrong answer with joint anchoring — and paid for it: AP fell from 1.0 to 0.948. The seesaw is real. r8 stays in the lineage as a documented negative result; r7 stays in production.
The invariant that matters more than any accuracy number: false negatives stayed at zero through every generation we shipped. A ranking gate that ever buries a real threat below the budget line is worse than no gate at all.
The trust contract: what the gate does not do
Most of Laya's design effort went into what it can't do:
- Reorder-only. The gate shuffles the verification queue. It never deletes a hypothesis. The original Scout set is preserved verbatim in every report we produce — the gate's reordered copy is an opinion layered on top, not a replacement.
- Fail-open. If the gate itself errors out, the pipeline falls back to the original order and notes the fallback in the report metadata. A broken gate degrades to part 4's behavior; it never blocks a scan.
- Minimal-information interface. The gate sees Scout-stage facts only — hypothesis text, file paths, agent0 baseline rows. Downstream verdicts never flow back into it. That is label-leakage hygiene, and it doubles as injection resistance: there is nothing in the gate's view for a crafted payload to hijack.
If you have read Anthropic's engineering posts on agent containment, the shape will look familiar: intercept, annotate, degrade gracefully — misfires must be survivable, because a gate that kills the session on every false positive makes long tasks impossible. Our version is narrower (we order work; we don't police actions), but the contract is the same genre.
The economics
Laya runs ~20 ms per hypothesis at steady state — well under one percent of what a single Verify pass costs. The gate's acceptance lineage went through three-arm chains (baseline / Laya / deterministic) with a pinned-hypothesis-set protocol, four acceptance criteria including confirmed-coverage, across two repositories and multiple environments. All of it is logged in a public eval ledger inside the repository, because a ranking claim without a reproducible trail is just a vibes statement.
The week it went live
This week we finally wired the gate into the production entry point — and, in the spirit of the series, the wiring itself was the first lesson.
During a false-positive calibration campaign on four real open-source repositories, we discovered the CLI had never actually attached the gate: the environment knobs were consumed by our eval harness, and the production entry point silently ran gate-less. The gates had passed every acceptance test — in the harness. The plumbing gap took an afternoon to fix and a regression test to lock, and the fix shipped the same day.
The first armed production run was everything we wanted: on one repository the gate reordered the Scout queue
[H2, H3, H1], flagged H1 as degenerate with a large negative score that sank it below the budget line, and reported fallback=False — the gate worked, and said so in metadata.Then the campaign itself: four real repositories (axios, pydantic, uv, pandas), seven semantic hypotheses total, zero confirmed findings, zero node failures. Seven candidate suspicions raised, all filtered by the verification layer before anything reached a report. Zero false positives — by construction there was nothing false to report. The raw artifacts (reports, summaries, logs) are committed in the repository under
reports/v12-fp/, because "zero findings" is exactly the claim that demands evidence.Appendix: the wall we hit first
The campaign almost didn't start. The first repository, axios, "finished" in ninety seconds with a nearly empty report — and we have been burned enough to know that fast-and-empty means something degraded.
The logs said it plainly: a 400 error — a request of 908,615 tokens against a 98,304-token context window. Scout had stuffed 446 code files into a single prompt. Our synthetic test repositories never had enough files to find the ceiling; the first real repository found it immediately.
The fix was governance, not a bigger window: a deterministic input budget (default 60K estimated tokens) that (1) always keeps AI-instruction files — AGENTS.md, CLAUDE.md,
.cursor/rules — since they are the highest-value audit surface, (2) fills small files first to maximize coverage, and (3) skips whole files over budget, with the skip count disclosed in the log. No half-file truncation: half a line of code is a hallucination generator. The rerun opened with the budget line in the log — 199 files / ~59,891 est. tokens (cap 60000); 247 files skipped over budget — and ran clean, end to end, zero degradation.What we still owe
Two items, already on the ledger:
- Scout sampling variance. The same repository produced one hypothesis in one run and three in another (the gate's temperature discipline applies to Verify; Scout still has its own). REFUTED-level noise is real and we want it measured, not hand-waved.
- gate v2: from ordering to skipping. Today the gate reorders within a budget; the next step is letting it decline to spend on hypotheses it ranks far below the line — with the skip disclosed exactly like the input budget above.
Close
Part 4 ended with a debt list. Part 5 pays the biggest item and adds two new ones, in public, with the receipts committed. That is the deal this series has been running on: everything we claim, you can pull and re-run.
The repository: a9320/arcanum-spark (MIT). The eval ledger lives at
docs/EVAL-LEDGER.md; this post's numbers trace to it. If you scan your own repository and disagree with a verdict — that is the most valuable issue you can file, and the template is waiting.Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article