AWS DevOps Agent: Beyond the Runbook
What does it look like when an autonomous agent investigates a real multi-region failure? I deployed a serverless payment ledger across Singapore and Tokyo, injected three simultaneous faults, and watched AWS DevOps Agent untangle all of them from a single alarm. This is the overview of the 2-part build series on dev.to.
Previously, I discussed how to leverage Amazon Bedrock to build a DR Toolkit , six AI-powered tools that automate the tedious parts of DR planning. That covered the prepare side. But preparation is only half the story. The other half is what happens during the incident itself: the moment your alarm fires, traffic is already rerouting, and someone still has to figure out what actually broke.
That gap felt like the next thing worth solving.
AWS DevOps Agent went GA back in March. It targets exactly that middle phase, so I built PayLedger, a multi-region serverless payment ledger, deployed it across Singapore and Tokyo with active-passive failover, and ran three real faults at it simultaneously to see what the agent would actually do.
The DR lifecycle gap
Most DR work happens at the edges. Here's what gets covered, and what doesn't:
| Phase | What happens | Covered by |
|---|---|---|
| Prepare | Runbooks, RTO/RPO targets, DR strategy, checklists | DR Toolkit |
| Detect | Alarm fires, SNS notifies DevOps Agent, health check fails, DNS fails over | CloudWatch + Route 53 + SNS |
| Investigate | Root cause analysis, cross-region signal correlation | ? |
| Recover | Apply fix, bring the unhealthy region back up, validate failback | Human + runbook |
| Learn | Prevention recommendations, operational improvements | DevOps Agent |
The Investigate phase is where the time actually goes. Alarms fire, Route 53 fails over automatically, but figuring out why a service running in the primary region is down, correlating signals across services, tracing the root cause, is still a manual job. That's exactly where AWS DevOps Agent is targeted.
What PayLedger does
PayLedger is a payment ledger. Four endpoints: record a transaction, list recent transactions, get the balance, and a health check. Deployed to ap-southeast-1 (Singapore, primary) and ap-northeast-1 (Tokyo, secondary) with Route 53 active-passive failover and DynamoDB Global Tables. A static Next.js frontend shows which region is serving: green for Singapore, amber for Tokyo.
When the primary goes down, Route 53 fails over within 20 seconds. The UI reflects the switch in real time. Users keep transacting. That's the failover story.
The investigation story is what runs in parallel.
The test: three faults at once
Instead of running one fault at a time, I injected all three simultaneously using
fault.py. One distinct fault per service:| # | Fault | How it breaks | Root cause category |
|---|---|---|---|
| 1 | IAM permission denied | Role swapped to fault role with no DynamoDB access | System change |
| 2 | Lambda throttling | Reserved concurrency = 0, 429 before function runs | Resource limits |
| 3 | Missing environment variable | TABLE_NAME removed, KeyError at module load | Code/config change |
One alarm fires. Three different root causes. Three different error signatures. The agent had to untangle all of them in a single run.
What the agent did
The investigation completed in 7 minutes 3 seconds. Triggered at 21:30:02, finished at 21:37:05. Within that window, the agent:
- Read its own context first. Before any AWS API call, the agent loaded the Agent Space Understanding skill and the PayLedger component reference. Both auto-generated from the connected account. By the 30-second mark, it already had architecture context.
- Split work across three parallel tracks. Lambda logs (11 tool calls), CloudTrail config changes (19 tool calls), and Lambda metrics (7 tool calls). Logs, audit trail, and metrics all running simultaneously instead of sequentially.
- Identified three distinct root causes. Missing
TABLE_NAMEcausing init crashes, wrong IAM role causingAccessDeniedException, and reserved concurrency = 0 causing throttles. - Reconciled every error. 100 5xx errors total: 90 from throttles, 5 from init crashes, 5 from IAM denials. The numbers added up exactly.
- Pinpointed the trigger. CloudTrail confirmed all three configuration changes happened within a 2-second window.
The agent's conclusion on the root cause:
"The role name 'payledger-dev-fault-iam', the use of Boto3 scripting, and the rapid self-recovery at 13:29:00Z strongly indicate this was a deliberate chaos engineering / fault injection exercise rather than an accidental misconfiguration."
It was right. The agent identified this as fault injection without being told.
The deeper build log
The full series on dev.to walks through both the architecture and the live demo:
- Part 1: Runbooks Don't Investigate. AWS DevOps Agent Does. The DR lifecycle gap, what AWS DevOps Agent is, and the multi-region PayLedger architecture
- Part 2: I Injected Three Faults. The Agent Found All of Them. Full setup walkthrough, the complete investigation timeline with actual log entries, and the honest assessment of what surprised me
You can find the full series here: DevOpsAgent: Beyond the Runbook .
DevOpsAgent: Beyond the Runbook Series
Try it or fork it
The PayLedger source is at github.com/romarcablao/payledger-aws-devops-agent . Fork it, deploy it, run the same fault injection script against your own Agent Space.
1
2
3
4
git clone https://github.com/romarcablao/payledger-aws-devops-agent.git
cd payledger-aws-devops-agent
./scripts/setup.sh
python scripts/fault.py inject
A few notes for SEA builders: AWS DevOps Agent is not available in ap-southeast-1 yet. The closest supported regions are ap-southeast-2 (Sydney) and ap-northeast-1 (Tokyo). Your workload can stay in Singapore. Only the Agent Space needs to be in a supported region, and cross-region monitoring covers the rest.
The full deploy guide is in Part 2 of the series.
References:
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article