Ground Truth: Trust Agent Claims by Provenance
Judge an AI agent's claims by their provenance, not their confidence: rank the source behind each claim and block anything that can't show its lineage.
Series: GenAI (3 articles)
- 2Ground Truth: Trust Agent Claims by Provenance This article
The problem: confident and wrong
If you have put an AI agent to work on real operations, such as triaging tickets, drafting runbooks, or summarizing an incident, you have watched it state something false with total confidence. Not a hedged guess: a crisp, well-formatted, authoritative sentence that happens to be wrong.
A wrong sentence in a chat window is harmless; you read it, correct it, and move on. The danger begins when the agent stops talking to you and starts writing to places other people trust: a comment on a ticket, a line in a change document, a step in a runbook an on-call engineer will follow at 2 a.m. without questioning it, because it is written down and it sounds sure. That is when a hallucination stops being a curiosity and becomes a liability.
We needed a way to let an agent write to shared surfaces without trusting it to police itself.
The insight: measure provenance, not confidence
The instinct is to ask the model how sure it is: a confidence score, "only act above 90%." This fails in the most dangerous direction. Language models are fluent by construction, and fluency reads as confidence. A model is often most fluent exactly when it is confabulating, because a fabricated detail carries none of the hesitation a real, messy fact does. Confidence measures how a sentence sounds, not whether it is true.
So we stopped scoring how sure the agent was and started scoring where each claim came from. Every claim has a source, whether or not it admits one. We rank those sources by their provenance, and let the provenance, not the phrasing, decide whether the claim can stand.
What provenance looks like
We classify the evidence behind a claim into a small hierarchy, strongest to weakest:
- Live observation: something read from a real system moments ago (a query result, a live status, an API response). Strongest.
- Authoritative reference: a fact from documented, maintained source-of-truth material.
- Recorded context: a value the agent read earlier in the same task.
- Model memory: something the agent "recalls" without re-checking.
- Model knowledge: a plausible-sounding fact from training, with no source at all. Weakest.
The class is the invariant; the specific evidence varies per claim. And the class is what gets scored. A beautifully worded claim resting on model knowledge is worth less than a blunt one backed by a live observation.
The rules that fall out
Once provenance is the unit of trust, the policy writes itself:
- A claim is only as strong as its weakest source. "The service is down and the cause is a bad config" is two claims. If the first came from a live check and the second from the model's intuition, the sentence isn't half-trustworthy; it's untrustworthy, because it hands a guess the same authority as a fact.
- Different claims demand different sources. "What is the state right now?" can only be answered by looking right now; a value from ten minutes ago may already be wrong. "What does this field mean?" must come from documentation, never recall. The required source shifts with the kind of claim: current state, count, timeline, identifier, root cause, and recommendation each carry a different bar.
- Higher stakes demand higher-grade sources. A scratch note tolerates a labeled guess. A permanent incident record or a change procedure does not. The same claim that is fine in a scratchpad is blocked from a document people will act on.
How the check runs: two tiers
Judging every claim with a large model on every write would be slow and expensive, so we use a cheap gate and an expensive gate, the shape many validation pipelines share.
- Deterministic gate (cheap, always runs). A code-only check with no model and no network verifies the mechanical, high-confidence cases: confirming, for example, that a claim referencing a documented structure matches that structure. It never hangs, never costs a token, and it is conservative by design. It fails a claim only when it can prove that claim wrong, so it never blocks honest work.
- Skeptical reviewer (expensive, when it matters). A second, deliberately adversarial model, separate from the one that wrote the text, reads the proposed output and the evidence behind it, classifies each claim's provenance, and scores it. The score is additive and auditable: it starts from a baseline and moves per claim according to the source behind it, so the final number is explainable, not a black-box verdict. Below a threshold, the reviewer blocks the write and names the specific ungrounded claims.
The reviewer's job is not to make the writing better. It is to make it honest. Those are different jobs, and conflating them produces prose that is more polished and no more true.
Where the checks live
We apply the same discipline everywhere our agents do consequential work: an agent that triages and resolves operational tickets before it posts findings, a tool that builds new agent skills before they ship, and the code-review gate they pass through on the way to production. Different surfaces, one rule: nothing factual reaches a place people rely on until its provenance is checked.
How it fits together
Trust is one stage in a larger loop where each stage feeds the next: mine improvement patterns from human work, build new skills from coverage gaps, refine them, and, before anything is promoted behind a human review, validate that what the agent claims is grounded in what it can show. Provenance checking is the trust layer of that loop. The companion piece, Skill Trials, covers a different layer of the same loop: how you make an agent better. This piece is about how you decide what to trust when it does.
What's novel
Most guardrails for agent output score the text (toxicity, format, a self-reported confidence) or run a second model to grade quality. Scoring provenance instead reframes the question from "does this read well, and does the model believe it?" to "can this claim point to a legitimate source, and is that source good enough for where it's being written?" Pairing a conservative deterministic gate with a separate, adversarial reviewer, and making the score additive and auditable instead of one opaque judgment, is the part we found most useful and least common in practice.
The part that humbled us
We built this trust layer to hold our agents to a higher standard, and then discovered the trust layer itself had shipped with a quiet flaw: in a common environment, the check silently reported everything as fine whether or not it was. The guard was waving traffic through, literally grading its own homework and giving itself an A.
The fix was the principle turned back on ourselves: stop trusting the check's self-report and demand evidence it does what it claims. We wrote tests that try to sneak fabricated facts past it and fail if any get through, and we made the honest, boring default ("I could not verify this") impossible to mistake for "this passed."
The discipline is fractal. The reason you don't trust an agent's confidence is the reason you don't trust a validator's green checkmark: a system reporting on itself will flatter itself. Anchor trust to evidence at every layer, including the layers you built to enforce it.
Try it yourself
The pattern is framework-agnostic:
- Stop scoring the agent's confidence; classify the source behind each claim.
- Rank sources: live observation beats documentation beats recall beats training memory.
- Write the two obvious rules: a claim is as strong as its weakest source, and higher-stakes surfaces demand higher-grade sources.
- Put a separate, skeptical reviewer between the agent and any shared surface; never let the author grade itself.
- Make the score additive and auditable so a block can name the exact ungrounded claim.
- Then point the same skepticism at your own guardrail, and test that it fails when it should.
You cannot make a language model stop being confident; that is the material you are building with. What you can do is refuse to let confidence stand in for truth.
Series: GenAI (3 articles)
- 2Ground Truth: Trust Agent Claims by Provenance This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article