Deterministic-First: GenAI Governance Without Hallucination

An architect pings an internal Slack bot: “Can I provision a t2.micro in production for the checkout service?”

A default Large Language Model (LLM) answers with confidence and invention. A conventional Retrieval-Augmented Generation (RAG) pipeline retrieves chunks of internal documentation, passes them to the model, and lets the model stitch an answer. Neither approach survives an audit.

Both answers will read plausibly. Neither will be governed. And in a regulated enterprise, plausible is not acceptable.

I spent an afternoon at the AWS São Paulo office during the AWS Brazil Well-Architected Hackathon building a prototype that solves this specific problem. Two days later I deployed the architecture end-to-end in my own AWS account to validate the design, funded the full test cycle at my own cost, then tore it down. The code is gone. The architectural thesis is what remains, and it is the only thing that matters.

I called the prototype Policy Guard. The codename is incidental. The pattern underneath is what I want to put on record.

The dominant architecture for enterprise GenAI assistants today follows a familiar recipe. Take the customer’s compliance documentation. Embed it into a vector store. When a user asks a question, retrieve the top-k chunks by semantic similarity, pass everything to a language model, let the model write the answer.

This is RAG, and it works beautifully for open-ended discovery. It fails structurally for compliance.

Three reasons.

First, semantic retrieval is not correct retrieval. Two chunks can be semantically close and factually contradictory. A chunk from the 2022 cost-optimization policy and a chunk from the 2024 revision will score nearly identically against a query about instance types. The model has no principled way to choose between them, so it blends. The blend is wrong.

Second, stitching is invention. When the model weaves an answer across three retrieved policy chunks, the output is a probabilistic narrative rather than a governed decision. The grammar is correct. The facts drift. The engineer reading the answer has no way to distinguish between what was in the source material and what the model generated in between.

Third, there is no audit trail. If an engineer asks whether a t2.micro is allowed in production and the model answers “yes, but consider upgrading to t3.micro for better performance,” which rule did it apply? Which version? Effective on what date? What happens when a compliance officer needs to reconstruct the decision six months later? A compliance-grade system answers every one of those questions. A RAG pipeline cannot.

This is not a bug. It is a limitation of the pattern. The pattern is wrong for the job, and no amount of prompt engineering fixes it.

Invert the stack. Make deterministic decisions first. Use the language model last, and constrain it tightly.

The model is not a reasoning engine for the decision. It is a natural language renderer operating over a validated JavaScript Object Notation (JSON) envelope. The decision happens before the model sees anything.

This is the line I want to put on record: choose where the model generates, not where it decides.

Once you internalize that separation, the architecture writes itself.

Policy Guard is a chat interface. Architects and engineers interact with it inside their existing tooling, which in practice means Slack or Microsoft Teams. Behind the interface, there are two planes that almost never touch each other.

The control plane is where rules are maintained. It ingests the customer’s internal compliance documentation on a schedule, normalizes it into structured rules, versions them, and writes them into an Amazon DynamoDB table. The same table also holds the complete set of Well-Architected Framework questions, grouped by pillar, as a foundational dataset. Every runtime decision cross-references the customer’s internal policy against the Well-Architected guidance. This plane is owned by the Cloud Center of Excellence (CCoE) or Platform team, and nothing in the runtime path can write to it.

The data plane is where queries are answered. It runs on AWS Step Functions Standard Workflows, orchestrating deterministic stages (intent classification, rule lookup, completeness check, Human-in-the-Loop when needed) and only then invokes a bounded large language model through Amazon Bedrock AgentCore. Every response carries a rule_id, a version number, and a pointer to the source document.

The separation is not aesthetic. It is an Identity and Access Management (IAM) boundary enforced at the role level. Writing a rule and answering a query are two different workloads with two different threat profiles, and they belong in two different planes.

Three architectural decisions define how the whole thing hangs together.

The challenge in a governance bot is that the source of truth lives outside the bot. Compliance teams maintain their policies in the customer’s internal documentation system. That system might be Confluence for one customer, Backstage for another, SharePoint for a third, or an internal content management system with a proprietary Application Programming Interface (API) for a fourth. The bot does not own that content. It consumes it.

The ingestion pipeline handles this. An Amazon EventBridge rule fires on a schedule. In my prototype the schedule was daily, but hourly works if the customer needs faster propagation. The scheduled event triggers an AWS Lambda function that pulls the latest documentation through a Model Context Protocol (MCP) server acting as a uniform interface to whatever back-end content system the customer happens to use.

MCP is the right abstraction for this layer. Customer documentation systems vary widely. The MCP server encapsulates the specifics of how to authenticate, paginate, and extract content from each one. The ingestion Lambda speaks MCP and does not care whether the documents live in Confluence, Backstage, or SharePoint. This is portability that matters in practice, because every enterprise eventually migrates their documentation system, and the bot should survive that migration without a rewrite.

The documents are not used directly. They are transformed. Each policy document is parsed into a structured rule with a rule_id, a version number, an effective_date, a changed_by field, and a schema describing the context the rule requires to be applied. A rule about instance types needs to know the environment (production, staging, development) and the workload category. A rule about data encryption needs to know whether the workload processes Personally Identifiable Information (PII). The schema is part of the rule, not hardcoded into the bot, because rules change faster than deployments do.

The structured rules are written to DynamoDB with version history preserved. The active version for each rule_id is a separate pointer, flipped atomically when a new version is promoted.

The DynamoDB table is not exclusive to customer rules. It also stores the complete set of Well-Architected Framework questions, grouped by the six pillars (operational excellence, security, reliability, performance efficiency, cost optimization, sustainability). The Well-Architected entries are loaded once as a foundational dataset and refreshed only when AWS publishes updates to the framework itself. This matters because every runtime decision is never based on a single source. The rule lookup pulls both the matching customer rule and the relevant Well-Architected questions for the domain, and both feed the completeness check and the envelope. The bot’s answer is simultaneously faithful to the customer’s internal policy and aligned with AWS best practice. That alignment is structural, not generative.

Ingestion is not real-time, and that is a deliberate choice. A real-time pipeline would couple the governance bot to the availability and latency of the customer’s documentation system. The scheduled pull model decouples them. If the documentation system is down for four hours, the bot keeps answering from the last known good snapshot. If the documentation system is slow, query latency does not degrade. The bot becomes more reliable than its own source of truth, which is the behavior you want in a governance system.

The trade-off is staleness. A rule change committed at ten in the morning will not propagate until the next scheduled pull. For compliance, that is acceptable. Compliance changes are announced in advance and governed by change management. If a customer needs minute-level freshness, they can trigger an on-demand pull through a webhook, but they rarely do in practice. The request is usually a symptom of something else, like a badly scoped ingestion window or a rule that should have been defined more broadly.

Rollback is a version pointer flip, not a redeploy. If a rule was wrong, flip the pointer and the next query uses the corrected one immediately.

Honest trade-off here: rollback is forward-only. Responses issued under the prior version are not retroactively invalidated. The audit log preserves the full history, so a regulator can always reconstruct which rule version generated which answer. In my experience, that is the trade-off every regulated environment accepts in practice.

The insight that made the rest of the architecture cheap is this: the state of the conversation is the execution of the workflow.

Most teams building chat assistants construct manual state machines on top of DynamoDB. They build polling Lambdas, reconstruct context on every turn, and end up with an improvised orchestration layer held together by cron jobs. I refused to do that.

AWS Step Functions Standard Workflows already provide everything needed. Durable state. Native timeouts up to one year. Retry and error handling declared in Amazon States Language (ASL). And the critical primitive for Human-in-the-Loop (HITL): waitForTaskToken.

The deterministic layer runs in stages inside the state machine.

An intent classifier uses regular expressions and weighted scoring to identify the policy domain. Cost, security, networking, availability, compliance. The domain determines which rule set to query. Classification is fast, explainable, and cheap. When it works, it works in tens of milliseconds. When it fails, it fails in a way that can be inspected and fixed, unlike an embedding-based classifier that gives you a score and no explanation.

A JSON lookup retrieves the relevant rule from DynamoDB using the active version pointer. A Choice state then evaluates completeness: does the query carry all the context required by the rule schema?

If the answer is no, the workflow does something that changes the economics of the entire system. It sends a parameterized follow-up to the user through Slack or Teams, generates a task token, and pauses. No compute. No charge for wait time. The user might answer in thirty seconds or three hours. When they do, a webhook calls SendTaskSuccess with the new context, and the execution resumes exactly where it stopped.

The follow-ups are deterministic. The questions are pulled directly from the Well-Architected entries in DynamoDB for the relevant policy domain, combined with the customer rule schema. The user never negotiates with a model about what information is missing. The machine knows what it needs and asks for it, because both the customer rule and the Well-Architected pillar tell it what to ask. “What environment is this for? Production, staging, or development?” “What is the expected throughput?” “Does this workload handle PII?”

Once the user provides the missing context, the workflow offers the candidate answer for acknowledgment. If the user accepts, the execution closes. If the user rejects, they explain why, and the explanation is captured in the workflow state. All of it, the original query, the follow-up questions, the user responses, the acceptance or rejection signal, the rejection reason, gets serialized into the JSON envelope that is eventually passed to the model.

That last detail matters. The user’s dissatisfaction becomes structured context, not unstructured complaint. The model does not have to guess why the previous answer was rejected. It is told, in fields with known names, and it renders the next answer accordingly.

Standard Workflows are required here, not Express. You need the full execution history for debugging, durable task token semantics, and an execution window long enough to accommodate conversations that pause overnight. Standard allows runs of up to one year, which is far more than this use case requires but comfortably covers the twenty-four hour timeout I configured as the cutoff for abandoned conversations. Express tops out at five minutes and keeps no history. For conversational state with HITL, Standard is the only correct answer. I see teams default to Express to save cost and regret the choice within the first quarter.

The model never receives a free-form prompt. It receives a JSON envelope with a strict schema. The user query. The matched rule with rule_id and version. The source document reference. The complete HITL context, including any previous acknowledgments or rejections. A set of output constraints describing what the response must include (the rule_id, the version, a link to the source) and what it must not include (speculation, legal advice, promises about third-party services).

Output is bounded generation. Around two hundred tokens. The model’s job is to render the decision that has already been made into polite, contextual English. It does not decide. It does not stitch. It does not browse.

The one place where the model does exercise judgment is disambiguation. When the intent classifier returns two domains with scores too close to separate deterministically, for example networking at 0.28 and security at 0.24, the model is asked to choose between them. Its output is validated against an allowlist of valid domain labels. If the model returns anything outside the allowlist, the pipeline falls back to a default safe response. The LLM is constrained even in the one place where it is allowed to think.

Guardrails are custom, not only the defaults. Personally Identifiable Information filtering, blocked topics, and a list of forbidden response patterns specific to the compliance domain. Amazon Bedrock Guardrails natively covers topics and PII. Output schema validation runs separately, because Guardrails is not a schema validator. The full stack that sits on the model output is: Guardrails for content, schema validator for structure, final envelope for delivery. Three layers, each responsible for one thing.

Collaborator agents under the Bedrock AgentCore Supervisor are routed by policy domain, but the routing decision itself takes the full deterministic context into account. Which follow-up questions were asked? What did the user reject in previous attempts? A user who already rejected a cost-optimization answer gets routed to the architecture collaborator next time, not back to cost. The router is aware of the conversation history that the deterministic layer already captured, because that history is in the envelope.

This is where the whole design becomes coherent. The model sees only what the deterministic layer decided it needed to see. Every decision downstream of the envelope is constrained by every decision upstream of it. Hallucination has no surface area.

What the docs don’t tell you

Things that only appear when you actually build this. waitForTaskToken cost model. You pay per state transition, not for the time the workflow is paused. A conversation that idles for six hours costs the same as one that completes in six minutes. This changes the economics of Human-in-the-Loop completely. Most teams discard HITL because they assume it is expensive. It is not, if you implement it this way.

Bedrock Guardrails is not a schema validator. It catches PII leakage and blocked topics. It does not enforce the structure of your output JSON. If the model returns a valid-sounding but schema-wrong response, Guardrails will let it through. You need a separate validation step that parses the model output against a declared schema and rejects anything that fails.

Cold start caches configuration. Warm Lambda drifts. The classifier patterns and the rule cache are loaded on cold start from Amazon Simple Storage Service (S3). If you change the configuration while Lambda containers are warm, they keep serving the old version until they get recycled. The fix is a Time To Live (TTL) on the cache. Short enough to converge, long enough to keep the hit rate high. For Policy Guard, sixty seconds was the right trade-off.

Standard Workflows versus Express. People default to Express for cost, and it is wrong for this use case. Standard allows executions of up to one year, preserves full history for debugging, and survives Lambda restarts without losing state. Express tops out at five minutes and keeps no history. For conversational state with HITL, Standard is the only correct answer.

MCP abstracts the customer’s documentation backend, not their semantics. The MCP layer gives you a uniform way to pull content from different systems. It does not give you a uniform way to interpret that content. Each customer’s policy format is different. The transformation from raw document to structured rule is customer-specific and unavoidable. Do not oversell this layer. MCP solves portability. It does not solve normalization.

Scheduled ingestion needs jitter. If every tenant schedules the ingestion Lambda at exactly two in the morning, the customer’s documentation API sees a thundering herd. Add a random jitter of zero to sixty seconds to the schedule per tenant. It costs nothing and protects the upstream.

The north star metric is queries-without-LLM-percentage. Observability that matters here is not p99 latency or error rate. It is the fraction of queries the deterministic layer resolves without invoking the model. When this number goes down, something regressed. A new policy domain was added without classifier patterns. Rules got too generic and the completeness check started failing too often. A collaborator was over-invoked. Track this metric above everything else.

Slack and Teams have incompatible webhook models. Slack expects responses within three seconds or it times out and retries. Teams is more forgiving but has stricter message formatting rules. The Step Functions state machine should emit an immediate acknowledgment (“working on it”) through the chat adapter before starting deterministic processing, regardless of which platform is being used. This is user experience (UX), not architecture, but it is the kind of detail that gets discovered on the first day of production usage and can force a redesign if you ignore it up front.

Use deterministic-first in regulated environments. Anywhere a compliance officer needs to answer “which rule generated this decision, in which version, on which date?” Anywhere a wrong answer has material consequences. Banks, insurers, healthcare, government, regulated industrial. Anywhere the auditor is a persona you actively design for.

Do not use it for brainstorming, creative exploration, or discovery-style retrieval. In those cases, the flexibility of the model is the product, and RAG is the right pattern. Deterministic-first trades flexibility for structural guarantee. That trade is only worth making when guarantees matter more than flexibility.

The inflection point is usually the first audit. Organizations that have never been audited tend to build with RAG and discover the problem when a regulator asks for proof. Organizations that have been audited already know the cost of a wrong answer and welcome the rigor from the start. If you are building for the first group, consider whether you are building for their present or their future.

Numbers from the prototype, validated in my own account and discarded.

Around eighty to ninety percent of queries are resolved entirely in the deterministic layer. Classifier plus rule lookup plus completeness check plus envelope generation, no language model invoked. Cost per query is fractions of a cent, dominated by Lambda invocation time measured in tens of milliseconds.

Of the remaining queries, most pass through the bounded LLM layer once, generate around two hundred tokens, and close. With current Bedrock pricing for mid-tier models, this is a few tenths of a cent per response.

Human-in-the-Loop conversations can span hours, but the economics do not degrade with duration. A conversation that waits for a user response overnight costs the same as one that resolves in two minutes, because the Step Functions execution is paused without billable compute during the wait.

Ingestion is effectively free at enterprise scale. A scheduled Lambda pulling through an MCP server once a day, processing a few hundred documents, is rounding error on an AWS bill.

Compared against a naive RAG plus LLM pipeline that invokes the model on every query with a generous context window, the cost reduction is between five and ten times. That is not the primary reason to adopt this pattern. Auditability is. But it is the reason a Chief Financial Officer approves the project on first pass.

Large language models are excellent generators. They are not reliable reasoners for compliance decisions.

Deterministic-first is not anti-LLM. It is pro-engineering.

Choose where the model generates, not where it decides.

Erick Mancz is an AWS Solutions Architect and AWS Golden Jacket holder. He writes Golden Jacket Field Notes weekly, covering ML/GenAI architecture on AWS from production trenches.