AWS Builder Center
Two Questions That Tell You Whether Your AI Architecture Can Survive an Audit

Two Questions That Tell You Whether Your AI Architecture Can Survive an Audit

Ask a document-chat tool how many records break a rule and it answers confidently from a ranked sample. Text-to-SQL fails the same way, invisibly. Adjudicated Query is an architecture pattern for questions where a missed record is a liability: exact filters fix the population, a rules engine makes the deterministic calls, and the model only routes questions and judges qualitative residue.

Adjudicated Query



An architecture pattern for answering natural-language questions when a missed record is a liability.

The need

An  organization holds a large body of records (leases, contracts, claims, credentials) that must satisfy externally-owned requirements, statutes, regulations, standards, which vary by category and change on someone else's schedule. At volume, automation replaces the human reviewer, and a new problem appears: the answer is now a number on a screen that nobody can independently check.
Two properties follow. First, provable completeness: "we checked all 22,910 of them" must be true and demonstrable. A record never assessed must be reported as unevaluated, never silently omitted. Second, defensibility: a finding may be challenged months later, so which rule version, which clause text, which method, and who signed off must all be recoverable.
Similarity-based retrieval (the RAG default) cannot satisfy either. Similarity has no threshold that means "all of them." A ranked sample never knows what it excluded. Text-to-SQL looks better on paper but fails invisibly: a hallucinated predicate silently narrows the population and the confident, exact-looking number is simply wrong.
Two questions expose this in any candidate architecture, before a line of code is written. What fixes the population? If the answer is similarity ranking, completeness is unprovable at any scale, not just at high volume. What makes the determination? If the answer is a model classifying everything, every verdict is an unrecorded opinion, including the ones arithmetic could have answered defensibly. An architecture can pass the first and fail the second.



The pattern: Adjudicated Query

A bounded conversational layer over a deterministic rules engine. The model does exactly two things: translate a question into a call on a fixed set of typed operations, and narrate the result that comes back. It never writes a query, never fixes the population, and never performs a determination.
Behind the boundary sits a deterministic engine. Rules are versioned data (never code). The engine knows generic operators:  gte ,  lte ,  equals ,  exists , and contains no branch naming a jurisdiction or topic. A law change is a rulebook edit, not a deploy.
Every determination records how it was made. Deterministic comparisons run first; the model is consulted only when a structured value is absent or the rule is genuinely qualitative. Findings are append-only. The completeness receipt is a computed invariant: compliant + in-breach + undecidable + not-evaluated = scanned, asserted before anything persists. A run that cannot account for its population never finishes.
The conversational surface carries counts, the receipt, and a labelled sample. The full result set lives on a tabular surface (a dashboard) reading the same store, drillable per record. This separation means the model never summarises away the guarantee.



Reference architecture

Reference Architecture

The chat agent reaches the engine through authenticated bounded operations. The dashboard queries the same store directly, read-only. Putting everything in one relational store makes the completeness receipt a SQL count, the cheapest possible way to make the central guarantee inspectable.
Two planes sit outside this view and a real build needs both: ingestion and extraction, which is what produces the unevaluated bucket, and a review surface where a human works the undecidable queue and their sign-off is recorded.



When to use it, and when not to

Use it when a missed record is a liability, answers may be challenged later, and the governing logic is externally owned. That covers lease compliance, insurance claims adjudication, sanctions screening, export control, credential verification, and clinical protocol monitoring.
Don't use it when plausible answers suffice (use RAG), when the question space is genuinely unbounded and a wrong answer is recoverable (use text-to-SQL), or when accountable users will tolerate dashboards without a conversational layer (use a rules engine and BI directly, same guarantee, less cost).
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article