The Night My Agents Went in Circles
A field note on AWS Step Functions, AgentCore, and what happens when no one's in charge
The pager went off at 3:11 a.m. on a Wednesday in early June. By 3:12 I was upright in the dark, squinting at a CloudWatch chart that looked like a heart-rate monitor having a panic attack. Eighteen hundred Bedrock invocations in nine minutes. Climbing. The dashboard's red threshold line had given up and walked off the screen.
I traced the calls. They formed a pattern I hadn't expected — not a runaway loop, not a fork bomb. A conversation. Two of my agents were talking to each other, in good faith, forever.
Triage had asked Research for context on a customer ticket. Research, finding the question slightly ambiguous, asked Triage to clarify what it meant by "context." Triage, eager to help, restated the original prompt with one extra word. Research, encouraged by the additional detail, asked another clarifying question. Round and round, every loop costing the same handful of cents in Bedrock tokens, every loop adding a polite paragraph to a transcript that no human would ever read.
It was the most well-mannered outage I have ever caused.
By 3:30 I had killed the deployment, throttled the provisioned model down to zero, and written one line in my Notes app: needs a conductor.
This is the story of finding one.
The architecture I had, drawn honestly
If I'd been asked at 2 a.m. what my multi-agent system looked like, I would have drawn confident arrows: triage routes to research, research returns to triage, triage calls escalation. Clean. Layered. Sensible.
The drawing I would have produced was a lie of omission. What I had actually built was a fully connected graph. Every agent could invoke every other agent through a thin Lambda. There was no central authority. There was no state machine. There was an implicit, unwritten hope that no agent would ever ask twice. Hope is a fine thing in life. It is not a control plane.
The fix was already sitting in the console, half-ignored, where it had been since 2016. AWS Step Functions. I'd used it before for ETL and never thought of it as something an agent system would need. Then on June 3, 2026 AWS shipped the optimized AgentCore harness integration, and I read the announcement on my phone over coffee on the morning after the incident, and the word that landed was harness. A harness is what you put on something that might otherwise wander off.
The shift in framing took me about a day. Stop thinking of agents as services that route to each other. Start thinking of them as specialists in a room with a conductor. The conductor decides who speaks next. The specialists do what they're best at, then return to silence. They don't get to invent edges in the graph. Only the conductor can do that. Only the state machine.
The first new wire
The first thing I rewrote was the triage flow. The Map state below fans out to specialists in parallel on AgentCore Runtime, then a Choice state routes on the merged result. The crucial change isn't visible in the code — it's what's missing. None of the agents call each other. They speak only to the conductor.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
{
"Comment": "Triage -> parallel specialists -> route on confidence -> done or escalate",
"StartAt": "Triage",
"States": {
"Triage": {
"Type": "Task",
"Resource": "arn:aws:states:::bedrockagentcore:invokeHarness",
"Parameters": {
"HarnessArn": "arn:aws:bedrock-agentcore:us-east-1:123456789012:harness/triage",
"Qualifier": "live",
"Payload": {
"input.$": "$.input"
}
},
"ResultPath": "$.triage",
"Retry": [{
"ErrorEquals": [
"BedrockAgentCore.ThrottlingException",
"States.TaskFailed"
],
"IntervalSeconds": 2,
"MaxAttempts": 4,
"BackoffRate": 2.0,
"JitterStrategy": "FULL"
}],
"Next": "FanOutSpecialists"
},
"FanOutSpecialists": {
"Type": "Map",
"MaxConcurrency": 4,
"ItemsPath": "$.triage.Payload.specialists",
"ItemSelector": {
"agent.$": "$$.Map.Item.Value",
"input.$": "$.input"
},
"ItemProcessor": {
"ProcessorConfig": { "Mode": "INLINE" },
"StartAt": "InvokeSpecialist",
"States": {
"InvokeSpecialist": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName": "agent-runner",
"Payload.$": "$"
},
"End": true
}
}
},
"ResultPath": "$.specialists",
"Next": "RouteOnConfidence"
},
"RouteOnConfidence": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.specialists[0].Payload.confidence",
"NumericGreaterThanEquals": 0.8,
"Next": "Done"
}
],
"Default": "EscalateToHuman"
},
"EscalateToHuman": {
"Type": "Task",
"Resource": "arn:aws:states:::sqs:sendMessage",
"Parameters": {
"QueueUrl": "https://sqs.us-east-1.amazonaws.com/123456789012/human-review",
"MessageBody.$": "$"
},
"End": true
},
"Done": { "Type": "Succeed" }
}
}A few notes from the trenches. The resource string is
arn:aws:states:::bedrockagentcore:invokeHarness — one word bedrockagentcore in the Step Functions URI, two words bedrock-agentcore in the harness ARN itself. I lost forty minutes the first day to that hyphen. The retry uses FULL jitter, not NONE, because when Bedrock throttles you, the worst thing you can do is retry in lockstep. The Map state's MaxConcurrency is 4, which feels timid until you've watched twenty parallel calls all earn throttling exceptions at the same instant.The second piece is the Lambda that wraps each specialist invocation. Step Functions Choice states can only branch on cleanly-shaped output, so the Lambda's job is to translate "whatever the agent returned" into a small, stable contract.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
import json
import os
import boto3
agentcore = boto3.client("bedrock-agentcore-runtime")
class AgentRetryable(Exception):
"""Raised for transient failures. Step Functions Retry block will back off."""
def handler(event, context):
harness_arn = event["agent"]["arn"]
user_input = event["input"]
try:
resp = agentcore.invoke_agent_runtime(
agentRuntimeArn=harness_arn,
qualifier="live",
payload=json.dumps({"input": user_input}).encode(),
)
body = json.loads(resp["response"].read())
except agentcore.exceptions.ThrottlingException as e:
# Surface as a typed error so Retry/Catch can act on the class.
raise AgentRetryable(f"ThrottlingException: {e}") from e
except agentcore.exceptions.ValidationException as e:
# Don't retry — the input is wrong. More attempts will not fix it.
return {
"status": "invalid",
"confidence": 0.0,
"answer": None,
"error": str(e),
}
# The contract Step Functions branches on. Keep it small. Keep it boring.
return {
"status": "ok",
"confidence": float(body.get("confidence", 0.0)),
"answer": body.get("answer"),
"next_hint": body.get("next_hint"), # advisory only — never authoritative
}next_hint is the line I underline for my team. The agent can suggest what should happen next. The state machine decides. That distinction is the entire bug, written backward. At 3 a.m. on the night of the incident, the agents had been allowed to choose their own next state. By the end of the rewrite, they couldn't.Four conductors, briefly compared
There are four shapes I see teams reach for when they need to stitch agents together. They are not interchangeable, and the wrong choice is expensive in different ways.
| Pattern | Durability | Observability | Cost shape | Reach for it when |
|---|---|---|---|---|
| Step Functions Standard | Execution history kept up to 1 year | Per-state input/output in console, CloudWatch, X-Ray | $0.025 per 1,000 transitions ($25 / million) | Long-running flows, human-in-the-loop, anything you'd want to replay or audit |
| Step Functions Express | In-memory; 5 minute max | CloudWatch logs only | $1.00 / million executions + duration | High-throughput, short, idempotent, no human gate |
| Bedrock Flows | Synchronous; no execution history | Limited tracing | Per node + model tokens | Pure prompt chains, no async, no approval steps |
| DIY Lambda fan-out | Whatever you build (usually: none) | Whatever you build | Lambda cost + your time | Prototypes on a Friday. Becomes a liability by Monday. |
The honest version of the last row: my pre-incident system was the bottom row. The polite-loop outage was that row collecting on its debt.
What bit me, and what I learned to look for
The integration only does Request-Response. As of June 22, 2026, the optimized AgentCore harness integration is in preview, available in four regions (us-east-1, us-west-2, eu-central-1, ap-southeast-2), and supports the Request-Response pattern only. There is no
.waitForTaskToken variant on the direct resource. If your agent needs to pause for a webhook, a human signature, or a callback hours later, you have two choices: wrap the call in a Lambda that publishes a task token and exits, or split the flow into two state machines stitched together by EventBridge. I burned a morning on this before I read the page carefully enough.256 KB stops being theoretical the moment transcripts grow. Step Functions caps state input and output at 256 KB combined. Roomy — until you start passing full conversation histories between states. My customer-support flow hit
States.DataLimitExceeded on the third turn, mid-transcript, in front of a stakeholder. The fix is to write large payloads to S3 once and pass an S3 reference between states. The lesson is to do this before you suspect you'll need it, not after.Concurrency and Bedrock throttling do not negotiate. I set
MaxConcurrency to 20 on the fan-out, felt very clever for about ninety seconds, and watched Bedrock throttle more than half the calls. The right number is not "as many as Step Functions will let me." It is "what my account-level model TPS will absorb, minus headroom for retries." MaxConcurrency: 4 with FULL jitter has carried me further than aggressive parallelism ever did. The instinct to parallelize harder is almost always wrong when the downstream is a model endpoint.What this changed
Two weeks in: zero loop incidents. The kind of zero where you stop checking.
Mean time to diagnose a failed run dropped from "open three dashboards and a transcript log and cry" to "click the red state in the console." Per-execution cost ticked up a few cents — Standard workflows are not free — and per-incident cost dropped to nothing, because there were no incidents.
The thing I didn't expect was that the agents got simpler. Once they stopped being routers, their prompts shortened. A specialist that doesn't have to decide "do I call someone else?" is easier to write, easier to evaluate, and easier to trust. The state machine absorbs the coordination so the agents can absorb the reasoning. That split — once you see it — is hard to unsee.
There's a sentence I keep coming back to. The state machine isn't the intelligence. The agents are. Step Functions is just the rule that says intelligence does not get to invent its own edges in the graph.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article