Governing Judgment, Not Just Output
A reframing of "reproducibility" for AI agents in policy-interpretation work, and a design pattern that bounds the process rather than the answer.
1. The wrong question
Imagine an audit engagement. A junior auditor is reviewing a client's revenue recognition for a multi-year software contract. The applicable standard runs to several hundred pages of human language and decades of interpretive guidance. The junior writes a memo: here are the facts, here is the standard, here is my conclusion. The partner reviews it.
The partner is not asking "is this answer correct?" — that question often has no clean answer. The partner is asking "is this answer defensible?" Did the junior identify the right unit of analysis? Did they consider the alternative interpretations a reasonable peer would consider? Did their judgment calls fall within a defensible range? Could a regulator looking at this engagement two years from now reconstruct why these conclusions were reached?
When we replace the junior with an agent, the conversation about AI quality almost always slides to the wrong question. We ask: can the agent get the right answer? We measure accuracy on benchmarks, hallucination rates on cited claims, agreement with reference outputs. These are useful proxies, but they miss what the partner actually does. The partner is not running an answer-key check. They are evaluating whether the work would survive challenge.
This article argues that for policy-interpretation tasks — auditing, regulatory compliance, professional advisory work — the design problem is not making the agent correct. It is making the agent's work legibly defensible, in the sense that a senior reviewer can efficiently verify, challenge, and sign off on what the agent did. That reframes "reproducibility" from a property of outputs into a property of process, and it changes what we should be building.
The piece focuses on one slice of a broader concept I'll call governed AI: the discipline of putting the right structural bounds around an AI system so its work is trustworthy in regulated, judgment-heavy contexts. Governed AI sits at three lenses — permissions, semantic context, and process bounds for judgment. The first two are more well-studied problem spaces. The third, where this article lives, is where the interesting and unsolved work is.
2. Why formalization is brittle for judgment
The most natural response to the problem above is formalization. If the policy is written in human language, translate it into machine-checkable rules. This pattern works well, sometimes brilliantly, on a specific shape of policy: the bright-line rule. "Revenue recognition requires the existence of a contract, identifiable performance obligations, a determinable transaction price, allocation to performance obligations, and recognition upon satisfaction." The structure is enumerable. The predicates can be checked. A neurosymbolic system can verify, with formal guarantees, that an answer satisfies them.
It works poorly on a different shape of policy, which is the shape that dominates real auditing standards: the judgment-mandating rule. "Management's estimate of fair value must be reasonable in light of the facts and circumstances." The standard is deliberately vague. It doesn't want to be made precise. The drafters knew that fixing a numeric threshold would produce both false confidence and gameable boundaries; they wrote it loose so that competent professionals could exercise judgment within it. Forcing this into formal predicates either requires the formalizer to invent a threshold the standard refuses to specify (introducing arbitrary precision), or produces a tautology ("the estimate is reasonable if it is reasonable") that catches nothing.
The brittleness, in other words, isn't a model capability problem that scales away with better translation. It is structural. The policy text resists formalization by design, because professional standards intentionally delegate judgment to qualified humans rather than fixing it ex ante. Formalization is the right tool for parts of these documents. It is the wrong tool for most of them.
This is not an argument against formalization. It is an argument that for the judgment-heavy parts of policy-interpretation work — which is where the agent's value and risk are concentrated — we need a different approach.
3. What we actually mean by reproducibility
Practitioners ask for "reproducibility" of AI systems and mean at least four different things. Untangling them matters because they require different mechanisms.
Determinism — same case in, same answer out. A technical property. Useful but largely solved by temperature-zero sampling, fixed seeds, and pinned model versions. Often what people say they want when they actually want something else.
Fidelity — the answer accurately reflects what the policy requires. A semantic property. This is what formalization tries to deliver, and where it succeeds for bright-line rules and breaks for judgment-mandating ones.
Justifiability — you can trace why the agent reached this answer. A process property. The agent could be wrong and the work still be justifiable, in the same way a junior auditor can give a wrong answer that survives review because their reasoning was reasonable given what they knew.
Robustness — trivial input perturbations don't flip the answer. A stability property. Important and underrated, but rarely the headline concern.
The audit partner's review is overwhelmingly a justifiability check. They are not running the case through a reference oracle. They are asking: did the work make sense? Was the right question framed? Were the right alternatives considered? Are the judgment calls within a defensible range? Could a regulator reconstruct the reasoning?
This matters for design because justifiability is the property formalization cannot deliver and that we are most often trying to fake with proxies. We measure hallucination rate as a proxy for fidelity. We measure self-consistency as a proxy for determinism. We measure benchmark accuracy as a proxy for both. None of these tell us whether the work is defensible. A confidently wrong agent with a fluent rationale will look excellent on most of these metrics and produce a workpaper that takes the partner three hours to discover is unsound.
Reframing the problem around justifiability changes what we build. Instead of optimizing for getting the right answer, we optimize for producing the right artifact: a record of what the agent considered, what it rejected, what it concluded, and how confident it was, structured such that a reviewer can navigate it efficiently.
4. Bound the process, not the output
The reframe in one sentence: the agent is a junior with perfect memory and no judgment, and the design move is making its homework legible, not constraining its conclusions.
A human junior brings fatigue, inconsistency, and limited capacity, but also a developed sense of when to pause, when to escalate, and what would catch a partner's attention. An agent inverts this: tireless and consistent, but with no internalized model of what the partner will challenge, and a strong tendency to produce fluent narrative that masks gaps in underlying reasoning. This inversion is the design constraint.
A junior's workpaper memo can be terse because the partner can fill in surrounding context — the junior's training, the engagement history, conversations during fieldwork. The partner reconstructs what the junior was thinking when something is unclear. With an agent, that reconstruction has no anchor. The agent's "thinking" wasn't witnessed and isn't reliable to introspect after the fact. Asking the agent post hoc "why did you conclude X?" produces a plausible narrative that may or may not reflect the actual computation.
The structural advantage of an agent over a junior is that it can make all of its context legible by construction. It can produce, alongside the answer, a complete record of the facts it gathered, the standards it consulted, the alternatives it weighed, and the rejections it made — without forgetting, without shortcutting, without retroactive narrative reconstruction. The catch is that the agent will not produce this record naturally. Left to its own devices, it produces a memo: a narrative output that reads well and selectively reveals reasoning.
The design move is to invert that default. Make the record the deliverable. Make the memo a generated view over the record. The agent's job is to populate a structured artifact whose shape forces it to externalize the things a partner needs to review. The narrative output exists, but it is downstream of the record and never the source of truth.
This invites the natural objection: doesn't structuring the output constrain the agent's reasoning? In practice, no — and this is the central design insight. The schema constrains what gets recorded, not how the agent thinks. The agent can reason freely about how to approach a novel case. What it cannot do is reach a conclusion without recording the facts that support it, the standards it relied on, the alternatives it considered, and its confidence in the result. Those are the things the partner needs. Demanding them does not constrain reasoning; it makes reasoning visible.
5. The shape of the artifact: a typed reasoning DAG
A reasoning record built for this purpose has three properties: it is typed (every piece of content has a known kind), it is graph-structured (relationships between pieces are explicit), and it is forensically stable (it remains interpretable months or years after creation).
The node types fall out of looking at how audit reasoning actually composes. Six categories show up repeatedly, with one important addition.
Fact. A claim about the entity under review, with a source citation. Distinct from interpretation. Facts are observed, not argued. "Revenue from contract X was recognized in fiscal Q3 totaling $4.2M" is a Fact. Its source is a workpaper extract, a system query, or a client representation.
Standard reference. A pin to a specific rule, version, and paragraph of the applicable policy. The rule text is captured at time of use rather than as a live link, because standards documents update and the record must remain interpretable as it was applied. "ASC 606-10-25-1 (as in effect 2026-05-15)" with the paragraph text inlined.
Interpretation. The agent's reading of how a Standard applies to a specific Fact pattern, anchored to the rule language it relies on. "The contract's renewal clause does not constitute a separate performance obligation under 606-10-25-19 because [...]". Judgment lives here.
Threshold or estimate. Any quantitative judgment — materiality, useful life, fair value range, confidence level — with its derivation, inputs, and sensitivity. Never a bare number. "Materiality threshold of $250K, derived from 5% of pretax income, sensitivity ±$40K under reasonable income assumptions."
Alternative considered. An explicit "I thought about doing X instead and rejected it because Y." Required for any non-trivial Interpretation, Threshold, or Conclusion. This is the schema's most opinionated demand: the agent cannot silently skip considerations because the structure refuses to be valid without them.
Conclusion. A terminal node tying the above together to support a determination, with a calibrated confidence value and an explicit comfort-zone flag (in / out of comfort zone, with reason if out).
Open question or reviewer escalation. A first-class node representing "I don't know" or "this requires human judgment." Crucial: without it, the agent will produce confident answers in places where it should be flagging uncertainty, because nothing in its incentive structure rewards uncertainty.
Composition is a directed acyclic graph. Each node has a stable identity. Edges between nodes are typed: a Conclusion is
supportedBy Facts and Interpretations; an Interpretation appliesTo Facts and anchors to a Standard; a Threshold is derivedFrom Facts and governedBy a Standard. This is a deliberate choice over two alternatives.A nested tree structure (each Conclusion contains its supporting nodes) duplicates supporting evidence when the same Fact feeds multiple Conclusions, which is the common case in audit work. A linear narrative with anchored references back to a node store reads more naturally but loses the structural advantages — you cannot filter, you cannot diff, and you cannot easily identify which nodes are load-bearing for which conclusions.
The DAG buys three things. First, every claim in the eventual narrative memo can hyperlink to its supporting node — review actions become navigation rather than reconstruction. Second, the structure is amenable to operations that get progressively important as the record graduates from internal review to regulator-grade evidence: serialization, hashing, signing, retention, comparison across cases. Third, the record can be filtered and queried — "show me every Threshold below 80% confidence", "show me every node tied to ASC 606-10-25-19" — without requiring a separate search index.
The captured-at-time-of-use rule text deserves emphasis. Standards update. Real audit work is sometimes reviewed years after completion. A record that links to "the current text of ASC 606" is forensically broken the moment the standard updates, because the record no longer reflects what the agent actually applied. Capturing the paragraph text into the Standard reference node, with a version stamp, makes the record stable. This is a small mechanical decision with disproportionate downstream value.
6. Adversarial validation: why structural checks aren't enough
A schema is only as strong as its validation. Structural validation — every Conclusion has a Standard, every Threshold has a derivation, every Interpretation cites rule text — catches the obvious failures. It does not catch the dangerous one.
The dangerous failure is a record that is structurally complete but substantively shallow. The agent produces an Alternative Considered node, but the alternative is a strawman the agent never seriously weighed. It cites a Standard, but not the right Standard for the specific issue. It produces a Threshold with a derivation, but the derivation rests on an assumption that nobody flagged. The shape of the record is correct. The substance is not.
Structural validation cannot reach this failure mode because the failure is in the content of the nodes, not their relationships. The record looks fine. A reviewer looking only at structure approves it. The error is found, if at all, when a different qualified reviewer reads it carefully — which is exactly the work the agent was supposed to compress.
The design response is adversarial completeness validation. After the agent produces a record, a second pass — distinct prompt, distinct context, possibly distinct model — attacks it. The adversary's job is not to produce a better answer. It is to surface what the original agent missed.
The adversary attacks four dimensions:
- Framing. Is this the relevant standard at all? Is this the right unit of analysis? Did the agent answer the right question? Many real audit failures aren't wrong answers to the right question; they're correct answers to the wrong question.
- Completeness. What standards should have been cited that weren't? What alternatives weren't considered? What evidence would change the conclusion if found?
- Conclusions. Where is the reasoning weakest? Which Interpretations or Thresholds wouldn't survive a competent reviewer's challenge?
- Citations. Do cited Facts actually support the claims attached to them? Do cited Standards say what the Interpretation claims they say?
Each surviving attack becomes an Open Question node appended to the record. The original agent must respond to each — either by extending the record (adding the missing Standard, Alternative, or evidence) or by escalating the question to the reviewer. The record is not "done" until the adversary produces no new substantive challenges within a budget.
Two design properties make this work better than it might appear.
The reviewer sees both the final record and the adversarial transcript. The transcript is itself diagnostic. A record that survived many rounds of meaningful challenge is more trustworthy than a record that cleared a single trivial pass. The kinds of challenges raised — and where the agent had to extend the record vs. defend it as written — reveal where the reasoning is fragile.
The framing attack is the most important and the most underrated. Most failures of expert reasoning aren't getting the answer wrong on a well-framed problem; they're framing the problem incorrectly and then competently answering the wrong question. An adversary that only attacks conclusions misses this entirely. Explicitly demanding that the adversary attack framing pulls the most dangerous failure mode into view.
The honest limitation: the adversary's quality bounds the validation's quality. Systematic blind spots in the adversary become systematic blind spots in the system. There is no clever architecture that fully closes this. The mitigation is calibration — periodically comparing agent records against reference records on cases with known-good worked examples — and rotation of adversary configuration over time. Calibration runs as a sampled background check, not per case. Its purpose is to detect when the adversary has stopped catching a class of error, not to validate every record.
7. The reviewer experience
The point of all of this is not the artifact for its own sake. It is to make the reviewer's four review actions cheap and reliable: evidence-to-conclusion traceability, completeness of consideration, judgment calibration, and standard-application correctness. The artifact pays off only insofar as it changes what review feels like.
The reviewer interacts with two views over the same underlying record.
The memo view is the default. It is a generated narrative resembling the workpaper memos reviewers already read, with each substantive claim hyperlinked to its supporting node. Confidence values and comfort-zone flags render inline so the reviewer cannot miss them. Open Questions and reviewer escalations surface prominently at the top, before the body — the reviewer sees what the agent flagged before reading the agent's confident statements. The adversarial transcript is collapsed but one click away.
Here's a snippet from an example work memo.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# WORKPAPER MEMO: SAAS CONTRACT REVENUE ACCOUNTING — ACME 36-MONTH SUBSCRIPTION
**Case ID:** saas-contract-2026-q1-customer-acme
---
## ⚠️ CRITICAL OPEN QUESTIONS — OUT OF COMFORT ZONE
The following issues must be resolved before final revenue figures can be confirmed:
### [oq1] Contract Value and Service-Period Arithmetic
[ref:oq1] The precise service-period denominator and total contract-value allocation remain unresolved. The contract states 36 months at $10,000/month = $360,000, but separately identifies $30,000 in implementation services. Three interpretations are plausible:
- (i) Total contract value is $390,000 ($360,000 subscription + $30,000 implementation separately priced);
- (ii) Subscription spans ~33–35 months at $10,000/month ($330,000–$350,000) plus $30,000 implementation = $360,000 total;
- (iii) Implementation is embedded in the $10,000/month rate.The memo-from-record direction is essential. Generating the record from a memo would let narrative fluency mask gaps in underlying reasoning, which is the failure mode this whole design exists to prevent. Generating the memo from the record makes the narrative a consequence of the structured reasoning rather than a substitute for it.
The record view is on demand. It is the DAG itself, navigable. Filterable by node type, by confidence value, by Standard reference, by adversarial-challenge survival. The reviewer goes here when they're spot-checking a specific Conclusion, looking for systematic patterns across the engagement, or challenging a particular judgment call.
Subset of the DAG for an example use case
The four review actions reduce as follows.
Traceability is now a click. Each claim in the memo links to its Fact node and its source citation. The reviewer no longer reconstructs which workpaper a claim came from; they navigate to it.
Completeness is now visible. Alternatives Considered are first-class nodes. The reviewer sees, at a glance, whether the agent considered a real alternative or a strawman, and the adversarial transcript shows what the adversary tried to add (and what the agent accepted vs. rejected).
Calibration is now structural. Every Threshold node carries its inputs, derivation, sensitivity, and rejected alternatives. The reviewer doesn't have to recall what a "reasonable" materiality threshold looks like in a vacuum; they see how it was computed and what would change it.
Standard-application correctness is now verifiable from the record itself. Version-pinned standard text is captured at the node. The reviewer doesn't have to remember what the standard said in this period; the record carries it.
Two further design moves matter for review effectiveness.
Annotation. The reviewer can mark any node — approve, reject, "push back here, regenerate," or free-form notes. Annotations serve two purposes. They are training and calibration signal that improves the system over time. They are also documentation that the reviewer engaged substantively with the work, which becomes load-bearing when the same artifact is later reviewed by a regulator. (A clean memo with no annotations is suspicious in regulator review; an annotated record demonstrates engagement.)
Diff view on regeneration. When a Conclusion is regenerated after a challenge or annotation, the reviewer sees a structural diff: which nodes changed, which were added, which were removed. This is how reviewers build intuition about which parts of the agent's reasoning are fragile and which are robust, without running A/B tests by hand.
The combined experience is closer to reviewing a junior who has prepared exceptionally well than to reviewing a black-box prediction. The reviewer's expertise is fully engaged on the things they're best at — challenging judgment, catching framing errors, exercising discretion. The mechanical work — checking citations, verifying versions, walking traces — is done by the artifact's structure rather than by the reviewer's recall.
Process flow
8. What this doesn't solve
Honesty about scope is part of the argument. The design above addresses one slice of governed AI. Several adjacent problems are out of scope, and the design works because it assumes they're handled elsewhere — not because it solves them.
Permissions and access (lens 1). This article assumes the agent only invokes tools and data sources that the current user and task are entitled to. That problem belongs to identity, least-privilege scoping, and policy engines for tool access. It is mature, well-understood, and unrelated to the reasoning record. If the agent reaches into data it shouldn't see, no schema fixes that.
Semantic context (lens 2). This article assumes the agent has structured context about what data means and how entities relate. If column names lie, if entity relationships are implicit, if business meaning lives only in tribal knowledge, the agent's Fact nodes will be subtly wrong in ways the schema cannot detect. A semantic layer or ontology is the right intervention there. It is a different sub-project.
Determinism. The design produces consistent records under consistent inputs through temperature-zero generation and pinned model versions, but determinism is not the goal. A different reasonable record is acceptable as long as it is justifiable. Treating "different output for same input" as a defect would force the system toward a brittleness it doesn't need.
Formal fidelity for judgment-mandating rules. The whole article is an argument that this is the wrong target. Bright-line rules can and should be formally checked; the design accommodates this as a complement to the reasoning record rather than a replacement. For judgment-mandating rules, formalization is the wrong tool, and no amount of better translation makes it the right one.
Review tiers beyond the senior reviewer. This design targets internal partner-tier review. Firm QA review, regulator inspection, and litigation defensibility have additional requirements — chain of custody, methodology compliance attestation, contemporaneous documentation discipline — that are not engineered here. The design is forward-compatible with those tiers (the DAG is signable, retainable, exportable) but does not solve them. Each tier deserves its own design pass.
Performance and cost. Adversarial validation roughly doubles inference cost per case at minimum, and a multi-round adversary plus regeneration loop can multiply it further. For some applications this envelope is fine; for others it is prohibitive. The design does not include cost optimization beyond a per-case budget on adversarial rounds. Real deployment requires explicit cost engineering.
The agent's own reasoning quality. The design constrains what gets recorded, not what gets reasoned. A weak underlying model produces weak Interpretations no matter how rigorous the record. The schema makes weakness visible; it does not eliminate it. This is the whole reason the reviewer remains essential.
Session-level temporal invariants. The design governs how a single conclusion is reasoned. It does not address path-dependent compliance — rules of the form "must notify supervisor within N steps of accessing X," or "authentication must precede database modification," or sequencing constraints across an entire engagement. Those are properties of the agent's trajectory over time, not of any single decision, and they need their own verification machinery (typically temporal-logic checkers running over the action stream). A complete governance posture would compose process-bound reasoning with trajectory verification; this article only does the former.
9. Open questions and research directions
The design is patterns-level and unvalidated. Several questions need real answers before — or alongside — implementation.
Node-type completeness. The seven node types come from looking at audit reasoning, but other policy-interpretation domains (regulatory compliance, legal analysis, clinical decision support, tax) likely have node types these miss. Is "analytical procedure" a distinct epistemic category in audit, or is it a Fact-with-derivation? Is "scoping decision" first-class? Empirical work with practitioners across domains would settle this; armchair design will not.
Adversary scope and budget. How many adversarial rounds are productive before diminishing returns? Does the adversary need domain specialization to produce useful challenges, or is a strong general-purpose model adequate? At what cost? The shape of the adversary is the largest open variable in the design, and the answer probably differs by domain.
Confidence calibration. Self-reported confidence is famously unreliable in language models. The design demands calibrated confidence on every Conclusion and Threshold; the question is how. Ensemble disagreement, perturbation tests, comparison against reference records, and explicit confidence training are all candidates. None is obviously dominant. This is one of the more methodologically interesting research questions in the design.
Standard text capture mechanics. The captured-at-time-of-use principle requires a versioned standards corpus, which most organizations do not have in machine-readable form. What's the practical path from "PDFs of standards" to "structured paragraph-addressable corpus with version stamps"? The unglamorous infrastructure work here is probably load-bearing.
Annotation feedback loop. Reviewer annotations are valuable signal. The risk is that the system silently learns to produce records that survive this reviewer's style rather than producing better records. How is reviewer signal incorporated without creating that dependency, and without the agent gaming the reviewer's known preferences?
Adversary failure modes. What happens when the adversary itself hallucinates a challenge? The original agent must distinguish "valid challenge I should respond to" from "spurious challenge I should reject," without sliding into either dismissive ignoring of real issues or endless spurious-loop responses. This needs careful design and probably needs reviewer-in-the-loop arbitration for borderline cases.
Cross-case consistency. The design is per-case. Real engagements have many related cases (multiple subsidiaries, multiple periods, multiple contracts) where consistency across the records matters as much as defensibility within each record. How does the system surface inconsistency across records, and how does it remediate it?
The line between formalization and judgment. The design assumes a clean split between bright-line rules (formalize) and judgment-mandating rules (record-and-validate). The split is rarely clean in practice — most rules have formal scaffolding around judgment cores. How does the system decide which posture to take for a given rule, and how does it handle rules that are partially formalizable?
Reducing formalization brittleness through redundancy. Recent work suggests that some of the brittleness of formalization comes from single-shot translation — one model converts policy text to formal rules, with no check that the translation is faithful. Multi-model approaches that translate the same policy through several independent models and verify logical equivalence (typically via SMT solvers) appear to substantially reduce false-positive translations. This is genuinely interesting because it strengthens, rather than contradicts, this article's thesis: formalization works for bright-line rules and can be made more reliable, while judgment-mandating rules still need process-bounding. Worth tracking as the formalization-as-component story matures.
Stochastic drift across long sessions. A concern adjacent to but outside this design's scope: in long-running sessions or multi-step engagements, agent behavior may degrade in compliance over time even if each individual decision is well-reasoned. Drift is a property of the session, not the decision, and may need session-level instruments — bounded recovery mechanisms, drift detection, periodic re-validation — to control. The reasoning record gives a per-case substrate for noticing drift (across-record comparison), but doesn't itself bound it.
Trajectory verification. Related to the temporal-invariants gap noted in scope: agent execution traces can be verified against temporal-logic specifications using established model-checking machinery. This is a tractable, well-studied problem space. A natural follow-on to the work in this article is composing the per-decision reasoning record with a per-session trajectory verifier — using the record's structured nodes (especially Conclusion and Open Question) as observable events that the verifier can reason about. We don't pursue it here, but the interface between the two is worth designing.
These are research directions, not gaps to be filled before shipping. Some will be answered by implementation; others by domain practitioners; others by adversarial pressure on real cases. Naming them explicitly is part of the argument: a design honest about what it doesn't yet know is more trustworthy than one that papers over its open questions.
10. Sample code
Sample code showing how to implement this approach will be available soon.
Disclaimer: I used an AI assistant to help research this article.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article