AWS Builder Center
6 lessons from building production multi-agent system on AWS

6 lessons from building production multi-agent system on AWS

This article provides the lesson learnt generalised from our cases and illustrate them with COBRA, a multi agent system built for quotations to our sales team.

AI Solution Architect | AWS CB | AI Tinkerer Hong Kong Co-Organiser | 22x AWS certified

Move rules out of the prompt: 6 lessons from a production multi-agent system on AWS

Based on my talk at AWS Community Day Hong Kong 2026 (24 October 2026, Level 300).
A prompt is a request. The model can ignore it. A guardrail is a control the model cannot get around.
Most of the work in building a multi-agent system that people trust is moving rules from the first kind to the second. This article walks through how we did that in COBRA, a multi-agent system I built for a sales team. It turns a free-text procurement email into a priced AWS Bill of Materials (BOM) and a shared AWS Pricing Calculator link. COBRA first ran in production on Amazon Quick Automate. We rebuilt it on Amazon Bedrock AgentCore and AWS Lambda, and that version is being cut over now.
The lessons are not specific to AWS. The examples are.

The problem: three things code alone could not solve

This is a typical request a sales rep forwards to us (illustrative, names changed):


Three phrases in that email explain why we needed agents and not just a script:
  1. "AWS calculator link": there is no API for it. The customer wants a public  calculator.aws  link. Only the Pricing Calculator web UI can create one. The Price List API returns prices, not links, and the newer Pricing Calculator API creates estimates inside an AWS account, shared through IAM, not public links. A fixed UI script breaks every time the page changes, so the pricing step is a browser agent that reads the page as it is.
  1. "DR" and "high security": requirements in free text. Sales reps do not know which AWS services these words mean, and fixed rules cannot read every way people write them. The database engine is not even stated.
  1. "Check it against AWS": documents to cross-check. An engineer used to read the customer's spec against AWS documentation line by line. That took about half a day per quote.

What it looks like for the user

The rep pastes the request into an Amazon Quick chat. COBRA asks only for what is missing, such as the payment option (pay as you go unless the rep says otherwise) and whether the app faces the internet. A fact with no safe default, such as the database engine, is a blocking question.


A few minutes later (about 5 in this run), the rep gets a BOM page by email. It shows the monthly cost, each assumption with its reason (for example, AWS WAF added because the app is public-facing), text ready to forward to the reseller, and the shared AWS Pricing Calculator link. Every line in the calculator is tagged with the BOM ID.


The demo is the easy part. The rest of this article is about what took longest to make trustworthy.

Architecture



The request path, left to right:
  • Amazon Quick chat, where the sales rep talks to COBRA.
  • Chat gateway: an  AgentCore Gateway  in front of the tools the chat can call. It uses OAuth with Amazon Cognito machine-to-machine (M2M) credentials, so the gateway authenticates the Quick connector, not the rep.
  • Core Lambda function: request intake and status lookups.
  • Orchestrator: a  Lambda durable function  that runs the stages in order. A durable function saves its progress after each step, can pause without compute charges while it waits, and can run for up to a year.
  • Pricing: a browser agent in its own  AgentCore Runtime , driving the Pricing Calculator in a Chromium that runs inside our own container. It calls back to the orchestrator when it finishes.
Supporting services: status and artefacts in Amazon DynamoDB and Amazon S3 under one AWS KMS key, Amazon Bedrock for the models, and a second gateway that accepts only IAM-signed calls for internal tools such as notifications (Amazon SES) and the cost estimator.
There are three trust zones: OAuth at the public edge, one IAM role per runtime, and internal tools behind IAM only. LLM tasks get no tools at all. Each returns JSON that code checks and then acts on. Even the browser agent only picks its next action from a fixed list; code runs it on the page and checks that the line was added. The model proposes, code acts.

Lesson 1: split where the output can be checked



COBRA has four agents. Each one is paired with a critic (we also call it an inspector): a separate check that reads the agent's output and returns pass or fail against written rules. Some critics are plain code, some are an LLM with code around it.
Agent
Produces
Its critic
Chat intake
a complete, valid request
request validator (code)
Requirements
typed needs, each quoting its source line
inspector: coverage and scope
Service mapping
AWS services, tiers and regions
inspector: catalogue and sizing
Pricing (browser agent)
a shared calculator estimate
calculator inspector, code first
Two choices matter here.
Coordination is code, not a manager agent. Stage order, retries and hand-offs are deterministic, so they live in the orchestrator. A manager agent would need every agent's context and would become the bottleneck.
Split where the output can be checked on its own, not by job title. Two examples from our migration:
  • The Quick Automate flow had LLM steps whose only job was to write fixed fields into a status record. There was nothing for a critic to judge, so they were not real split points. They became plain code.
  • Large requests kept failing at the Requirements stage. We split the input into chunks that a critic can check one at a time, and read the chunks in parallel.
The strip at the bottom of the diagram is the rule behind the whole design: put each rule in the hardest layer that can enforce it. Here is one real COBRA rule at each layer, from weakest to strongest:
Layer
What enforces it
A COBRA rule at this layer
Prompt
the model, if it follows the instruction
the browser agent may visit  calculator.aws  only (a known gap, see Lesson 5)
Schema
the model cannot generate output outside the schema
the chosen service must be one value from our service catalogue
Critic
a second check, often an LLM
every sentence in the request is covered by a need, or labelled as not a requirement
Code
the same answer every time
an operating system must be quoted from the request, or it is  not_stated  (Lesson 2)
Orchestration
the workflow, not any agent
3 stage attempts, then a person; only code changes the BOM status
Identity
IAM, outside the application
each runtime has its own IAM role; internal tools accept only IAM-signed calls
A critic sits below code because an LLM critic can be wrong (Lesson 3 has an example). Code returns the same answer for the same input. When a rule can be written as code, write it as code.

Lesson 2: one contract for every model call



Every LLM call in COBRA goes through the same function:
  1. Bedrock Converse with the JSON schema enforced at decode time ( structured output  through  outputConfig ). "At decode time" means the model can only generate tokens that fit the schema, so the shape is right even when the content is wrong, unless the answer is cut off at the token cap. Code still checks the schema. We do not rely on forced tool choice to shape output. One model ignored it when the input conflicted with the schema.
  1. A validator in code runs the schema, Python business rules, and a grounding check: the text the model quotes must really appear in the source.
  1. Errors go back to the model as a re-ask, each listed by JSON path, for at most 2 repairs. After that, keep the items that pass and drop the rest ("salvage"), or fail the step.
  1. Hitting the output token cap fails at once. There is nothing useful to repair.
  1. If fewer than 90 seconds remain in the Lambda invocation, stop and retry in a fresh invocation.
  1. Validated answers are cached per step in S3. When a durable function resumes, it replays: it runs the code again from the start and skips the steps it has already finished. The cache covers a step that runs again, after a timeout or a crash, so a model call that already passed is not made or paid for twice.
The core of that function, simplified (throttling retries, the cache, the time check and salvage are left out):
import json
import boto3
import jsonschema
from botocore.config import Config

bedrock = boto3.client("bedrock-runtime", config=Config(read_timeout=300)) # long answers outlast the 60 s default

class BudgetExceeded(Exception): ...
class SchemaInvalid(Exception): ...

def llm_call(task, model, system, user, schema, validate=None, max_repairs=2, max_tokens=6000):
"""One narrow task: schema enforced at decode time, then code validation, then at most 2 repairs."""
validator = jsonschema.Draft202012Validator(schema)
messages = [{"role": "user", "content": [{"text": user}]}]
for _ in range(max_repairs + 1):
resp = bedrock.converse(
modelId=model, system=[{"text": system}], messages=messages,
inferenceConfig={"maxTokens": max_tokens},
outputConfig={"textFormat": {"type": "json_schema", "structure": {"jsonSchema": {
"schema": json.dumps(schema), "name": "result", "description": "result"}}}})
if resp["stopReason"] == "max_tokens":
raise BudgetExceeded(task) # a cut-off answer: nothing to repair
text = "".join(c.get("text", "") for c in resp["output"]["message"]["content"])
try:
obj = json.loads(text)
errors = [f"{'/'.join(map(str, e.absolute_path)) or '(root)'}: {e.message}"
for e in validator.iter_errors(obj)]
except json.JSONDecodeError as e:
obj, errors = None, [f"(root): not valid JSON ({e.msg})"]
if not errors and validate:
errors = validate(obj) # business rules and grounding, in code
if not errors:
return obj
messages += [{"role": "assistant", "content": [{"text": text}]},
{"role": "user", "content": [{"text": "Your answer has these problems. "
"Return the full corrected JSON.\n- " + "\n- ".join(errors)}]}]
raise SchemaInvalid(f"{task}: {errors[:3]}")
Each task passes its own  validate  function. That function is where most guardrails live.

Schema rules are a test, not a style guide

We learnt three schema rules the hard way, and a unit test fails if any of the main task schemas breaks them:
  • Every array has  maxItems . Without it, one model looped until it hit the token cap. Support for schema keywords differs by model, so a model goes on our allow-list only after we have called it with structured output enforced.
  • No numeric bounds ( minimum ,  maximum ) in the schema. The  Bedrock structured output docs  list them as unsupported. One model accepted them anyway and emitted whitespace until the cap. Bounds are checked in code instead.
  • The quote field comes before the decision field. We ask for the source text first. The order is not guaranteed (one model wrote the quote after the decision), so code always checks the decision against the quote.
def test_every_model_schema_follows_the_contract():
seen = collect_task_schemas() # [(task, schema)]: runs the main LLM tasks with llm_call patched to record them

def walk(node, path):
if isinstance(node, dict):
assert not {"minimum", "maximum", "exclusiveMinimum", "exclusiveMaximum"} & set(node), path
if node.get("type") == "array":
assert "maxItems" in node, path
for k, v in node.items():
walk(v, f"{path}/{k}")
elif isinstance(node, list):
for v in node:
walk(v, path)

for task, schema in seen:
walk(schema, task)

# in each extracted need, the quoted words come before the value decided from them
req = next(s for t, s in seen if t == "extract_needs")["properties"]["needs"]["items"]["required"]
for quote, decision in (("quantityText", "quantity"), ("osText", "osFamily"),
("dbEngineText", "dbEngine"), ("deploymentText", "deployment")):
assert req.index(quote) < req.index(decision)
If a rule is not a failing test, it is only a suggestion.

A sample guardrail: code verifies, it never extracts

The quote fields pay off in the validator. For each need the requirements agent extracts, code checks that the quoted words really are in the customer's line, that every number is written in its quoted evidence, and that a decision such as the operating system has words behind it (simplified from our code):
def need_errors(n):
"""Code checks one typed need. Each error goes back to the model with its JSON path."""
errs, line = [], n["sourceLine"]
for f in ("osText", "dbEngineText"):
if n[f] and not grounded(n[f], line):
errs.append(f"{f}: must be words from sourceLine")
for k, m in enumerate(n["measures"]):
if not grounded(m["evidence"], line):
errs.append(f"measures/{k}/evidence: must be words from sourceLine")
if m["value"] not in numbers_in(m["evidence"]):
errs.append(f"measures/{k}/value: {m['value']:g} is not written in its evidence \"{m['evidence']}\"")
if n["osFamily"] != "not_stated" and not n["osText"]:
errs.append("osFamily: quote the words naming the operating system in osText, or use not_stated")
if n["dbEngine"] != "not_stated" and not n["dbEngineText"]:
errs.append("dbEngine: quote the words naming the engine in dbEngineText, or use not_stated")
return errs
The last two checks enforce a rule that used to be a line in the prompt: "never assume Linux". If the request does not name an operating system, the model must answer  not_stated , and the rep is asked, in the chat or later on the BOM page. COBRA never guesses the OS. The only mapping to "Linux" is in the calculator step, for named distributions such as Ubuntu or Rocky Linux, because the calculator has no option for them.

Lesson 3: the critic checks with code first



The reflection pattern (a generator paired with a critic) is the technique that helped us most. But a critic that is only another LLM asked "is this good?" tends to agree with everything, or fail everything.
After pricing, the calculator inspector reads the live shared calculator link (never an export) and checks each line:
  • Code checks first: a line is missing or duplicated, the region is wrong, server settings differ from the plan, or the cost is more than 5 times off an estimate from the AWS Price List API.
  • The LLM judges only settings that change the price. A setting the request does not state, a default, or an extra at quantity zero is not a failure. Without this rule the inspector failed everything and the stage retried until its budget ran out.
  • Code sets the verdict. A failed line is corrected in the calculator: edited in place, or added again if it is missing or duplicated. If the model fails a line whose cost already matches the list-price estimate, the plan itself may be wrong, so the line goes to our team instead.
Critics hallucinate too. Our LLM inspector once failed a pricing line with this reason: "She was the first woman to fly solo across the Atlantic." The fix was in code: a claimed mismatch must share words with the request, or it is sent back.
def ungrounded(requested, line):
"""True when the 'requested' side of a claimed mismatch shares no word with the line's request or plan."""
claim = words(requested) # lower-case words, stop words removed
source = words(json.dumps({k: line.get(k) for k in ("service", "request", "cobraSettings")}))
return bool(claim) and not (claim & source)

def validate(obj):
errors = []
for n, x in enumerate(obj["lines"]):
x["verdict"] = "fail" if x["mismatches"] else "pass" # code sets the verdict, not the model
if x["verdict"] == "fail" and not x["fix"].strip():
errors.append(f"lines/{n}: a fail needs its mismatches and a fix")
for k, m in enumerate(x["mismatches"]):
if ungrounded(m["requested"], lines_by_id[x["itemId"]]):
errors.append(f"lines/{n}/mismatches/{k}/requested: quote what the line's request "
f"or COBRA's settings say; {m['requested'][:60]!r} is not in them")
return errors
For a request line "1 database server, 8 vCPU, 32 GB RAM, SQL Server Standard, 500 GB data":
  • A mismatch with requested "SQL Server Standard" and page "MySQL" shares the words "sql", "server" and "standard" with the request. It is kept, and the line fails.
  • A mismatch with requested "She was the first woman to fly solo across the Atlantic" shares no words with it. It goes back to the model as an error at its JSON path.

Lesson 4: every retry has a budget, then a person



Unbounded self-correction is a bug. Every loop in COBRA has a limit:
  • Model call: the durable step runs up to 3 times on throttling, 5xx errors and timeouts. Each run also retries the Bedrock call with its own backoff.
  • Schema repair: 2, then the attempt fails.
  • Stage: at most 3 attempts.
  • Then a person: a blocking question to the rep, or a hand-off to our team with the failed stage shown as the BOM status (for example, "AWS Service Mapping Failed").
We used to switch to a larger model (Kimi K3) on the last attempt. We removed it. Whole-stage reruns on the bigger model cost a lot and rarely passed. When the third try fails, the input is usually missing something, and a person can supply it.
Some things are never the model's decision:
  • Status changes: code, with versioned writes.
  • Emails: each status email has an idempotency key, built from the BOM, its version, the round and the email type. The sender records the key before sending and skips any key it has already recorded, so a retry never sends the same status email twice.
  • Hand-offs between agents: artefacts stored by key in S3, never chat history.
A whole round is capped at 6 hours by the durable function's execution timeout, with a sweep every 10 minutes as a second check.

Lesson 5: know what is your architecture and what is your platform

COBRA's orchestrator has run on three AWS runtimes. The design stayed the same each time. What we had to build ourselves changed.

Step Functions (v1 to v4)
AgentCore Runtime (v5)
Lambda durable functions (now)
LLM repair loop
Amazon States Language (ASL) states, awkward
our code
our code, checkpointed steps
Waiting for pricing
callbacks, awkward
8-hour async cap (microVM)
callback, no compute charge while paused
State between steps
payload limits
we built checkpoints
built in
Crash recovery
built in
we built lease, heartbeat, sweep
replay; LLM steps at most once per retry
Longest run
1 year
8 hours
1 year (we cap at 6 hours)
Moving to durable functions deleted our lease, heartbeat and resume code. Our rule of thumb: long waits plus an LLM repair loop point to durable functions.

Pricing: our own runtime, our own browser

Pricing runs in its own AgentCore Runtime: a container image with a browser, permissions only it has, and runs that can last hours.
We started with  AgentCore Browser , a managed browser that our code drove remotely over the Chrome DevTools Protocol (CDP). For our workload, each in-page call took 5 to 15 seconds, sessions froze often, the calculator's dropdowns (built with Cloudscape, the AWS UI library) closed under remote control, and a tab crashed on a 12-item estimate. Chromium inside our own container is now the default, and the remote browser is a configuration switch. In both modes, actions run as in-page JavaScript, a 30-second script timeout triggers a reload, and a run resumes from its checkpoint up to twice.
Running our own browser means we own patching and network egress (which sites the container can reach). A known gap: "go to  calculator.aws  only" is still a prompt rule today, not a network control. That is exactly the kind of rule this article says to move down a layer. Egress control is the fix.

Quick Automate vs AgentCore: a tool vs a system


Amazon Quick Automate (v1, the prototype)
Amazon Bedrock AgentCore (now)
What it gives you
drag and drop, built-in connectors, nothing to run or patch
code: agents, tools and orchestration in Git
Best for
one person automating their own work
a system other people depend on
Who changes it
anyone, in the flow editor
developers, and AI coding agents
How you test
sample requests run by hand: hours a round
unit, offline and end-to-end tests: minutes a round
What you own
very little
orchestration, state, gateways, identity
Quick Automate was the right place to prototype. We switched mainly for testing. On Quick Automate, every change meant running sample requests by hand and reading each stage's output. In code, the test suites run in minutes, and an AI coding agent (Claude Code, in our case) can write a change and run those tests itself. That only works when the system is code.
What carried over unchanged: the split into agents, a critic for each producer, and the retry budgets. The architecture moved from one platform to the other without changing.

Lesson 6: fix the harness before the model



By harness we mean everything around the model: the prompts, schemas, validators, tools, data and tests. When an LLM step fails, the cause is usually in the harness, not the model.
Our tests, from fast to slow:
  • About 167 unit tests, many named after the real BOM that broke.
  • A schema contract test (Lesson 2).
  • An offline harness that runs the stages with real Bedrock calls and an in-memory store.
  • End-to-end tests on the dev environment, through the chat gateway, that check the email and the BOM page.
  • A browser-agent bench: 25 hand-written cases plus one per service in our catalogue. A case passes only if the line exists in the calculator and the inspector passes it.
  • Evals (full runs scored against known good answers): 26 earlier BOMs, compared by hand today. Running them in CI is planned.
When a real BOM breaks, it becomes a test with that BOM's name.
Each of these failures was first blamed on the model:
Symptom (we blamed the model)
Real cause
Fix in the harness
"4 vCPU / 16 GB" read as 16 GB of disk
code re-parsed the wording with regular expressions
the model returns typed needs with the quoted source; code only verifies, and a lint stops the regex parsers coming back
AgentCore Gateway priced as Amazon Bedrock
the service was not in our catalogue, so the tool could not do the job
108 services added; code rejects any service other than the one the line names
Large requests failed at Requirements
the output token budget
read in chunks, in parallel; keep what passes instead of failing the stage
Amazon S3 picked for an SMB file share
enum values had no descriptions
a one-line description for every option
A better model getting past a gap in the harness is not evidence that the model was the problem. Only change a task's model when you have shown the model is the cause.

Pick the model on its worst test, not its mean



We ran 8 COBRA-shaped tests, 5 runs each, through Bedrock Converse with schema output (October 2026). The score is the share of checks passed. The means all look fine. The worst tests do not: three models scored 0 on at least one task, which means that task is broken for them.
Then we put price next to quality. These are on-demand list prices in Asia Pacific (Tokyo), from the AWS Price List API in October 2026. Kimi K3 and Grok 4.6 show the price of their global cross-Region inference profiles, which is how we tested them:
Model
Worst test
Input, USD per 1M tokens
Output, USD per 1M tokens
Kimi K3 (global)
0.83
3.00
15.00
Kimi K2 Thinking
0.75
0.73
3.03
Grok 4.6 (global)
0.65
2.00
6.00
Qwen3 235B
0.50
0.27
1.06
DeepSeek V3.2
0
0.74
2.22
MiniMax M2.5
0
0.36
1.44
Devstral 2
0
0.48
2.40
Qwen3 235B is the cheapest of the seven and has no broken task. Kimi K2 Thinking does better on its worst test, but costs about 3 times as much and was about 4 times slower per call in our tests (16 seconds against 4). Kimi K3 does best, at about 11 times the price for input and 14 times for output. Qwen3's weakest test also showed us where the harness needed work, which is cheaper to fix than paying for a larger model on every call. So every model call in COBRA uses Qwen3 235B. That includes the browser agent, which reads the calculator page as text (its accessibility tree), not as screenshots.

What is still open

  • Browser egress is a prompt rule, not a network control (see Lesson 5).
  • Tracing: each call is logged with the BOM ID, validated outputs are in S3, and the browser logs steps and screenshots. There is no end-to-end OpenTelemetry tracing across Lambda, AgentCore Gateway and AgentCore Runtime yet.
  • Cost per BOM: model tokens are logged for every call, but browser and runtime time are not measured per BOM yet.
  • Evals are compared by hand. Moving them into CI is next.

Takeaways

  1. Put each rule in the hardest layer that can enforce it. Prompt, schema, critic, code, orchestration, identity.
  1. Schemas shape the output; code decides.
  1. Every producer gets a critic, and the critic is mostly code.
  1. Budget every retry, then hand to a person.
  1. Know what you own and what the platform gives you.
  1. Fix the harness before the model.

Where to start on your own system

  1. Put every model call through one function. Schema enforced at decode time, a validator in code, at most 2 repairs, and a hard failure on the output cap.
  1. Add the schema contract test.  maxItems  on every array, no numeric bounds, quote before decision.
  1. Give one critic code checks first. Pick the agent whose mistakes cost most, and check what code can check before any LLM judges it.
  1. Write down a retry budget for every loop, and decide which person gets the work when the budget runs out.
  1. Turn each production failure into a test named after the request that broke, and fix the harness before you change the model.


John Ng is an AI and Cloud Solution Architect in Hong Kong, an AWS Community Builder, and a co-organiser of AI Tinkerers Hong Kong. If you are putting agents on a tool with no API, or want to compare notes on the architecture or the test harness,  connect with me on LinkedIn .
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article