When Agents Disagree: Multi-Agent Failure Modes on AWS.
Adding a second agent turns your system into a distributed one that happens to reason. Part 9 of Production Agents covers the failure modes that live between agents: cascading errors, orchestration loops, duplicate actions, and silent drops. Grounded on Bedrock AgentCore and Step Functions, with six containment patterns (timeouts, idempotency keys, circuit breakers, and more).
This series has spent eight parts making a single agent trustworthy. We put one
on the factory floor, moved it from demo to deployment, wrapped it in guardrails,
built an evaluation harness to prove it works, counted its tokens, gave it memory,
and put a human in the loop. Every one of those posts assumed a single agent, or a
single agent with tools.
Then you add a second agent, and a third, and hand them a shared goal. Suddenly
the failure modes are not about one agent being wrong. They are about agents being
wrong at each other: a coordinator that loops, two workers that both act on the
same task, a sub-agent that silently drops the ball while the orchestrator reports
success. These bugs are harder to see because no single component looks broken.
The system is broken in the spaces between the components.
This is the topic I kept deferring. Here it is.
The failure taxonomy
Four patterns account for most multi-agent incidents I have seen.
Cascading failure. A sub-agent returns a plausible but wrong result. The
orchestrator, having no reason to doubt it, builds on it. By the time the error
surfaces, three agents have compounded it and the root cause is buried. Single-agent
eval does not catch this because each agent, in isolation, behaved reasonably.
Orchestration loops and deadlock. Agent A asks Agent B to clarify, B asks A to
clarify, and they trade turns until you run out of budget. Or a coordinator keeps
re-delegating the same task because it never recognizes the work as done. The
system is busy and produces nothing.
Conflicting or duplicate actions. Two agents independently decide to create the
same ticket, send the same email, or refund the same order. Nothing errored. You
just did the thing twice, and one of those was a side effect you cannot take back.
Silent drops. A sub-agent times out or returns empty. The orchestrator's prompt
does not account for the empty case, so it proceeds as if the step succeeded. The
worst kind of failure: a confident final answer built on a missing middle.
Grounding it on AWS
The AWS building blocks map onto containment naturally, which is why this series
has leaned on them throughout.
Use Amazon Bedrock AgentCore for the agents themselves and for the primitives that
make coordination observable: memory for shared state, and the runtime hooks you
have seen in earlier parts. For the coordination fabric, reach for AWS Step
Functions rather than letting one agent free-form orchestrate the others. A state
machine gives you something a prompt cannot: explicit timeouts per state, retry
policies with backoff, catch blocks for the failure branch, and a visual execution
history that shows exactly where a run stalled. When agents coordinate through a
state machine instead of through each other's imaginations, most of the loop and
deadlock class simply cannot happen.
Containment patterns
Detection is not enough. You need the system to fail safely on its own. Six
patterns, roughly in order of value.
Per-hop timeouts. Every agent-to-agent call gets a deadline. A silent drop
becomes an explicit timeout you can branch on. In Step Functions this is a state
field; in code it is a hard wall around every sub-agent invocation. No unbounded
waits, ever.
Idempotency keys. Any action with a side effect (create, send, refund) carries
a key derived from the task, not from the agent. If two agents attempt the same
action, the second is a no-op. This alone kills the duplicate-action class.
Circuit breakers. Track sub-agent failure rates. When one crosses a threshold,
stop calling it and fall back to a degraded but safe path rather than letting it poison every downstream step. A flaky agent should get benched, not kept in the
game.
An arbitration agent. When two agents disagree, do not average their answers.
Route the conflict to a dedicated referee with a narrow, well-tested prompt whose
only job is to pick, and to escalate to a human when it cannot. Disagreement
becomes a decision point, not a coin flip.
Dead-letter handling. A task that fails every retry goes to a dead-letter queue
for human review instead of vanishing. You want a place where lost work
accumulates visibly, not silently.
Loop budgets. Cap total delegations per run. When the budget is spent, halt and
escalate. A system that has taken twenty hops without finishing is not thinking, it
is stuck.
Tie it back to the series
The earlier parts are not separate from this, they are the instruments.
- The evaluation harness from Part 5 is how you detect these failures before production. Write eval cases where a sub-agent returns wrong or empty, and assert the system contains it rather than propagating it. Test the seams, not just the agents.
- The guardrails from Part 3 are how you bound the actions each agent can take, so a confused agent cannot reach for something dangerous.
- The memory from Part 7 is how agents share state deliberately through a store you control, rather than by passing brittle context blobs and hoping.
A multi-agent system is not a smarter single agent. It is a distributed system that
happens to reason, and it deserves the same discipline you would give any
distributed system: timeouts, idempotency, circuit breakers, dead letters, and a
clear escalation path. The intelligence is the easy part now. The engineering
around the intelligence is where production agents are won or lost.
What to build first
Do not try to add all six patterns at once. Start with the two that pay for
themselves immediately: per-hop timeouts, so silent drops become visible, and
idempotency keys, so duplicate actions stop being possible. Add an eval case for
each. Once those hold, layer in circuit breakers and an arbitration path. By the
time you reach dead letters and loop budgets, you will have a system that fails in
ways you can see and recover from, which is the whole game.
Next in the series: prompt-injection across agent boundaries, where one agent's
output is another agent's untrusted input
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article