AgentOps Chaos Lab: When AI Agents Become Your Incident Response Team

Learn how to build an AI-driven chaos engineering platform where multiple specialized agents collaborate to investigate deliberately injected failures, correlate evidence across AWS services, and produce actionable recovery plans transforming incident response from reactive firefighting into orchestrated intelligence.

Series: AgentOps on AWS: From Chaos to Intelligent Recovery (2 articles)

1
AWS DevOps Agent Challenge: What I Learned from Infrastructure Failures
2
AgentOps Chaos Lab: When AI Agents Become Your Incident Response Team This article

Dineshraj Dhanapathy

Sharing, Learning, Growing Together

Published Jul 5, 2026

Last modified Jul 5, 2026

The Problem
Solution Overview
How the Workflow Operates in Practice
Architecture
Technical Deep Dive
- Chaos Engine
- Signal Monitoring and Correlation
- Multi-Agent Orchestration
- MCP Server Integration
- Slack Notifications
- Runbook-Driven Recovery
Deployment Steps
Testing
DevOps Agent Integration
Extending the Pattern
Conclusion

The Problem

When cascading failures hit production infrastructure, the typical incident response follows a painful pattern. A CloudWatch alarm fires. An on-call engineer wakes up, opens a laptop, and begins context-switching between CloudWatch metrics, CloudTrail events, ECS service consoles, Lambda logs, and IAM policies trying to piece together what broke and why.

The challenge is not identifying that something is wrong. CloudWatch alarms handle detection reliably. The challenge is correlation. A single misconfigured IAM policy does not produce a single error. It triggers retry storms in Lambda, connection timeouts in ECS, cascading 5XX responses in API Gateway, dead-letter queue overflow in SQS, and misleading alarm fatigue in CloudWatch all simultaneously. The on-call engineer must mentally correlate dozens of signals across six or seven AWS services to identify one root cause buried under layers of symptoms.

The core issues are clear: single-point investigation tools cannot handle multi-service correlation, human engineers cannot simultaneously analyze CloudWatch, CloudTrail, ECS, Lambda, IAM, and Secrets Manager in real-time, and the quality of incident response depends entirely on who happens to be on-call and how deeply they understand each affected service.

What if instead of one overloaded engineer, you had a team of six specialists each an expert in their domain collaborating to investigate failures simultaneously? And what if you could deliberately inject those failures to validate your investigation capabilities before a real incident occurs?

Solution Overview

AgentOps Chaos Lab combines two powerful concepts: chaos engineering (deliberately breaking infrastructure to validate resilience) and agentic AI (multiple specialized agents collaborating on complex tasks). The result is a platform that injects failures, detects symptoms, dispatches domain-specific investigation agents, correlates their findings, and produces unified recovery plans autonomously.

The system operates across three layers. The Chaos Engine maintains a library of 16 injectable failure types across 6 categories and can inject them into real AWS infrastructure (Lambda, IAM, Secrets Manager, CloudWatch) or simulate them locally. The Detection Layer monitors CloudWatch alarms, CloudTrail events, ECS service health, and Lambda metrics, correlating raw signals into structured incidents. The Investigation Layer dispatches six specialized AI agents Infrastructure, Security, Cost, Deployment, MCP Health, and Recovery each analyzing the incident from their domain expertise and contributing findings to a unified root cause report.

The platform validates its own investigation capabilities by comparing multi-agent findings against the AWS DevOps Agent, which independently queries real AWS telemetry. This dual-validation approach ensures the system produces accurate root cause analysis.

How the Workflow Operates in Practice

The autonomous investigation pipeline handles any failure that can be detected by AWS monitoring services. The core principle: when something breaks, the system follows the exact diagnostic steps your best engineers would follow across multiple domains simultaneously, correlating findings that no single engineer could synthesize as quickly.

Consider a cascading failure scenario at 3 AM. A bad deployment pushes a non-existent container image tag to an ECS task definition. The MCP server fails to start. All AI agent tool calls begin returning ConnectionRefusedError. Meanwhile, an IAM policy change removes Bedrock access, and someone manually opens SSH to the world on a security group. Three independent failures, all happening within minutes of each other.

A human on-call engineer would spend 45 minutes switching between the ECS console, CloudTrail, IAM policies, and security group rules before understanding the full picture. The AgentOps Chaos Lab identifies all three root causes, their interdependencies, and a prioritized recovery plan in under 30 seconds.

The workflow follows a five-phase execution model:

Phase 1: Injection. The Chaos Engine selects one or more failures from its library and injects them either simulated (for development) or live (actually modifying AWS resources). Each failure has defined expected symptoms, affected services, and recovery steps.

Phase 2: Detection. The Chaos Monitor scans CloudWatch alarms (state changes), CloudWatch Logs (error patterns like ConnectionRefusedError, AccessDeniedException), CloudTrail (failed API calls), ECS events (task failures, deployment issues), and Lambda metrics (Duration, Errors, Throttles). Raw signals are collected with metadata including source, severity, timestamp, and related resources.

Phase 3: Correlation. Signals are grouped by time proximity and resource relationships, then correlated into incidents. The monitor infers the likely failure category (MCP, IAM, deployment, data, networking, or agent failures) and assigns a confidence score based on signal density and consistency.

Phase 4: Investigation. The Orchestrator receives the correlated incident and dispatches relevant specialized agents based on the failure category. Each agent investigates from its domain perspective, produces structured findings with evidence, severity, confidence, and recommendations. Findings are passed forward later agents benefit from earlier discoveries.

Phase 5: Recovery. The Recovery Agent synthesizes all findings into a prioritized, dependency-aware recovery plan. Steps that unblock other steps come first. Each step includes the exact CLI commands, verification steps, and estimated time.

Architecture

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
┌─────────────────────────────────────────────────────────────────────┐
│                     YOUR AWS INFRASTRUCTURE                           │
│   Lambda  │  CloudWatch  │  CloudTrail  │  IAM  │  Secrets Manager  │
│           │              │              │       │                    │
│   Emits metrics, logs, and API events continuously                  │
└──────────────────────────────┬──────────────────────────────────────┘
                               │ Metrics / Events
                               ▼
┌─────────────────────────────────────────────────────────────────────┐
│                         CHAOS ENGINE                                  │
│  ┌────────────┐  ┌────────────┐  ┌────────────┐  ┌──────────────┐  │
│  │  Failure   │  │  Injector  │  │  Monitor   │  │     CLI      │  │
│  │  Library   │  │ (16 types) │  │(CW/CT/ECS) │  │ (Rich output)│  │
│  │  6 cats    │  │ sim + live │  │ correlate  │  │              │  │
│  └────────────┘  └─────┬──────┘  └─────┬──────┘  └──────────────┘  │
└─────────────────────────┼───────────────┼───────────────────────────┘
              Injects │           Detects │
                      ▼                   ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    PLANNER / ORCHESTRATOR                             │
│         Receives incident • Dispatches agents by category            │
│         Correlates findings • Produces unified report                │
└──────┬──────┬──────┬──────┬──────┬──────┬───────────────────────────┘
       │      │      │      │      │      │
       ▼      ▼      ▼      ▼      ▼      ▼
┌────────┐┌────────┐┌────────┐┌────────┐┌────────┐┌────────┐
│Infra   ││Security││ Cost   ││Deploy  ││  MCP   ││Recovery│
│Agent   ││Agent   ││Agent   ││Agent   ││Health  ││Agent   │
│        ││        ││        ││        ││Agent   ││        │
│ECS,    ││IAM,    ││Waste,  ││CFN,    ││Connect ││Plan,   │
│Lambda, ││Secrets ││Budget, ││Images, ││Schema, ││Steps,  │
│Network ││Trail   ││Optimize││Artifact││Context ││Verify  │
└────────┘└────────┘└────────┘└────────┘└────────┘└────────┘
       │      │      │      │      │      │
       └──────┴──────┴──────┴──────┴──────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    UNIFIED ROOT CAUSE REPORT                          │
│    Root cause + confidence + cascading effects + recovery plan       │
│    Markdown / JSON / HTML with timeline + prevention recs            │
└──────────────────────────────┬──────────────────────────────────────┘
                               │
                    ┌──────────┼──────────┐
                    ▼          ▼          ▼
              ┌──────────┐ ┌──────┐ ┌──────────┐
              │  Slack   │ │Report│ │  DevOps  │
              │  Alert   │ │ File │ │  Agent   │
              │          │ │      │ │(validate)│
              └──────────┘ └──────┘ └──────────┘

Core Concepts

Concept	Description
Failure Library	16 injectable failure types with defined symptoms, recovery steps, and cascading effects
Chaos Injector	Executes failures in simulated (safe) or live (real AWS) mode with rollback support
Chaos Monitor	Multi-source signal scanner that correlates raw alerts into structured incidents
Orchestrator	Dispatches specialized agents based on failure category and aggregates findings
Specialized Agents	Domain experts (Infrastructure, Security, Cost, Deployment, MCP Health, Recovery)
Unified Report	Correlated root cause with prioritized recovery plan and prevention recommendations
MCP Server	Exposes chaos lab tools to AI agents via Model Context Protocol (JSON-RPC)

Technical Deep Dive

Chaos Engine

The Chaos Engine is the foundation of the platform. It maintains a comprehensive library of failures that represent real-world production incidents, organized into six categories.

MCP Failures cover scenarios specific to agentic AI systems: broken server connections, tool schema version mismatches, and agent memory corruption. These are increasingly relevant as organizations deploy AI agents that depend on MCP servers for tool access.

IAM Failures cover credential and access management: missing Secrets Manager secrets and expired IAM credentials. These create widespread blast radius because every AWS service call depends on valid credentials.

Deployment Failures cover release pipeline issues: CloudFormation drift (manual changes creating state inconsistency), corrupted deployment artifacts, and wrong container image tags. These are the most common cause of production incidents in practice.

Data/AI Failures cover the AI-specific infrastructure: vector database unavailability, embedding model dimension mismatches, and Bedrock model access revocation. As AI systems become production-critical, these failure modes need dedicated investigation expertise.

Networking Failures cover service communication: API Gateway integration failures, disabled EventBridge rules, and dead-letter queue overflow. These often manifest as "everything is broken" because they disrupt inter-service communication.

Agent Failures cover the investigation system itself: context corruption, hallucinated resource references, and circular reasoning loops. Meta-failures where the investigation tool itself is compromised.

Each failure in the library defines:

1
2
3
4
5
6
7
8
9
10
11
12
13
@dataclass

class Failure:

    id: str                          # e.g., "lambda-001"

    name: str                        # e.g., "Lambda Timeout (CPU Starvation)"

    category: FailureCategory        # Which category it belongs to

    severity: Severity               # low, medium, high, critical

    description: str                 # What happens when injected

    injection_method: str            # How to inject it

    expected_symptoms: list[str]     # What signals to expect

    recovery_steps: list[str]        # How to fix it

    aws_services_affected: list[str] # Blast radius

    cascading_effects: list[str]     # What else breaks

    detection_signals: list[str]     # How monitoring detects it

The Injector supports two execution modes. Simulated mode logs what would happen without touching real resources — ideal for development, demos, and testing the agent pipeline. Live mode actually modifies AWS resources using boto3: updating Lambda function code, modifying IAM policies, deleting secrets, and misconfiguring alarms. Every live injection records rollback information so the system can restore clean state.

1
2
3
4
5
6
7
8
9
10
injector = ChaosInjector(mode="live")

# Single failure
result = injector.inject_random(category=FailureCategory.IAM_FAILURES)

# Cascading scenario (3 simultaneous failures)
results = injector.inject_cascade(count=3)

# Undo everything
injector.rollback_all()

Signal Monitoring and Correlation

After injection, the Chaos Monitor scans five AWS signal sources:

CloudWatch Alarms - The monitor calls describe_alarms(StateValue="ALARM") to find all alarms currently firing. Each alarm in ALARM state becomes a signal with its namespace, metric, threshold, and current value.

CloudWatch Logs Insights - The monitor searches log groups for error patterns using filter_log_events. It looks for ConnectionRefusedError, AccessDeniedException, TimeoutError, ResourceNotFoundException, and "Task timed out" across agent and Lambda log groups.

CloudTrail - The monitor calls lookup_events filtering for write operations with error codes. This captures failed API calls like InvokeModel returning AccessDeniedException, or PutRolePolicy events that indicate IAM changes.

ECS Events - The monitor checks describe_services for any service where runningCount < desiredCount. This detects container failures, image pull errors, and health check failures.

Lambda Metrics - The monitor queries get_metric_statistics for Errors and Duration metrics on chaos lab functions, identifying timeout and invocation failure patterns

Raw signals are then correlated into incidents:

1
2
3
4
5
6
7
8
9
10
11
signals = monitor.scan(lookback_minutes=15)  # Returns 5-50 raw signals
incidents = monitor.correlate()               # Groups into 1-3 incidents

# Each incident has:
# - incident_id: "INC-0001"
# - detected_at: timestamp
# - signals: list of correlated signals
# - affected_services: ["ECS", "Lambda", "API Gateway", "Bedrock"]
# - likely_category: "mcp_failures"
# - confidence: 0.85
# - summary: human-readable description

The correlation engine groups signals by time proximity (5-minute windows), extracts affected AWS services from resource ARNs, infers the failure category from signal patterns (e.g., ConnectionRefused + ECS 0 tasks → mcp_failures), and calculates confidence based on signal density (more correlated signals = higher confidence).

Multi-Agent Orchestration

The Orchestrator is the central coordinator. When it receives a correlated incident, it makes two decisions: which agents to dispatch and in what order.

Agent dispatch is determined by failure category:

Failure Category	Agents Dispatched
`mcp_failures`	MCP Health → Infrastructure → Recovery
`iam_failures`	Security → Infrastructure → Recovery
`deployment_failures`	Deployment → Infrastructure → Recovery
`data_failures`	Infrastructure → MCP Health → Recovery
`networking_failures`	Infrastructure → Deployment → Recovery
`agent_failures`	MCP Health → Security → Recovery
`unknown`	All agents

Two rules govern dispatch: the Cost Agent is always added for critical-severity incidents (financial impact must be quantified), and the Recovery Agent always runs last (it needs findings from all other agents to build a dependency-aware plan).

Each agent receives the incident context plus accumulated findings from prior agents. This is critical - the Recovery Agent doesn't independently discover root causes; it synthesizes findings from Infrastructure, Security, and Deployment agents into a coherent plan:

1
2
3
4
5
6
7
8
9
for agent_name in plan.priority_order:

agent = self._agents[agent_name]



# Each agent gets the incident + all prior findings

context = dict(incident)

context["prior_findings"] = [r.to_dict() for r in agent_reports]



report = agent.investigate(context)

agent_reports.append(report)

Every agent produces structured findings:

1
2
3
4
5
6
7
8
9
10
11
AgentFinding(

    agent_name="Security Agent",

    category="iam_failure",

    severity="critical",

    title="Widespread AccessDenied - Bedrock InvokeModel",

    description="47 AccessDeniedException events from chaos-lab-execution-role...",

    evidence=[{"type": "cloudtrail", "data": {"event": "InvokeModel", "error": "AccessDeniedException", "count": 47}}],

    affected_resources=["arn:aws:iam::466742534146:role/chaos-lab-execution-role"],

    recommendations=["Restore bedrock:InvokeModel permission", "Implement IAM policy change alerting"],

    confidence=0.95,

)

The Orchestrator then correlates all findings by severity and confidence to determine the root cause, identifies cascading effects, builds a prioritized recovery plan, and renders the unified report in Markdown, JSON, or HTML.

MCP Server Integration

The Chaos Lab exposes its capabilities via Model Context Protocol (MCP), making it accessible to any MCP-compatible AI agent. The MCP server implements protocol version 2024-11-05 over stdio transport using JSON-RPC.

Eight tools are exposed:

Tool	Purpose	Autonomy
chaos_inject_failure	Inject a specific failure by ID	Requires confirmation
chaos_inject_random	Random injection with optional category filter	Requires confirmation
chaos_monitor_scan	Scan for active signals across all sources	Fully autonomous
chaos_get_incidents	Get correlated incidents	Fully autonomous
chaos_investigate	Trigger full multi-agent investigation	Fully autonomous
chaos_list_failures	List available failure types	Fully autonomous
chaos_rollback	Undo all injected failures	Requires confirmation
chaos_get_report	Get latest investigation report	Fully autonomous

The MCP server separates read-only operations (scan, list, report) from mutating operations (inject, rollback). In production deployments, mutating tools would require human approval — similar to the approval workflow described in the AWS DevOps Agent's MCP pattern.

{"method": "tools/call", "params": {"name": "chaos_inject_failure", "arguments": {"failure_id": "lambda-001", "mode": "simulated"}}}

The server handles tool routing, error handling, and response formatting. Any MCP-compatible agent (including AWS DevOps Agent with a custom MCP server registration) can call these tools to orchestrate chaos testing programmatically.

Slack Notifications

Every lifecycle event in the chaos lab generates a Slack notification using Block Kit formatting for rich, actionable messages:

Injection Alert - Posted immediately when a failure is injected. Includes failure name, ID, mode, timestamp, and expected symptoms. The on-call team knows something was deliberately broken and what symptoms to expect.

Incident Detection - Posted when the monitor correlates signals into an incident. Includes incident ID, category, signal count, confidence, and affected services. Includes interactive buttons for "Investigate" and "Acknowledge."

Investigation Complete - Posted when the multi-agent investigation finishes. Includes root cause with confidence, investigation duration, and the top 5 recovery steps. This is the most valuable notification - the team gets a complete diagnosis without opening any console.

Recovery Confirmation - Posted after rollback or manual fix. Confirms how many failures were rolled back and the current system status.

1
2
3
4
5
notifier = SlackNotifier(webhook_url=os.environ["SLACK_WEBHOOK_URL"])

notifier.notify_injection(injection_result)

notifier.notify_incident(incident)

notifier.notify_investigation_complete(report)

notifier.notify_recovery(rollback_status)

Runbook-Driven Recovery

The platform includes four operational runbooks that define step-by-step recovery procedures:

MCP Server Recovery - Handles ECS task failures, image pull errors, and service health check failures. Steps include verifying task definition, checking ECR for valid tags, updating task definition, forcing new deployment, and verifying health endpoint.

Lambda Timeout Recovery - Handles injected delays and CPU starvation. Steps include confirming timeout via metrics, checking recent code changes in CloudTrail, reviewing function code for sleep patterns, redeploying clean code, and verifying invocation success.

IAM Credential Recovery - Handles AccessDeniedException and expired credentials. Steps include identifying the affected principal via CloudTrail, checking role policy documents, restoring permissions, forcing credential refresh, and monitoring for continued failures.

CloudFormation Drift Remediation - Handles manual changes outside IaC. Steps include initiating drift detection, reviewing each drifted resource, deciding revert vs import, applying fixes, and verifying clean state.

Each runbook includes detection criteria, severity assessment, step-by-step commands, verification steps, and prevention recommendations. The Recovery Agent references these runbooks when constructing its prioritized recovery plan.

Deployment Steps

Prerequisites
AWS CLI configured with admin access
Python 3.11+ with boto3 installed
AWS account in us-east-1 (free-tier eligible resources)
Step 1: Deploy Infrastructure
The minimal stack creates Lambda functions, CloudWatch alarms, IAM roles, a Secrets Manager secret, and a CloudWatch dashboard all free-tier eligible:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
cd chaos-lab/infra
aws cloudformation deploy 
--template-file deploy-minimal.yaml 
--stack-name chaos-lab-minimal 
--capabilities CAPABILITY_NAMED_IAM 
--region us-east-1
Wait for: Successfully created/updated stack - chaos-lab-minimal (2-3 minutes)
What this creates:
Lambda chaos-lab-handler (5s timeout, 256MB)
Lambda chaos-lab-cpu-spike (30s timeout, 512MB)
IAM Role chaos-lab-execution-role (Bedrock + CloudWatch + Logs)
3 CloudWatch Alarms (Duration, Errors, Throttles)
1 CloudWatch Dashboard (ChaosLab-Overview)
1 Secrets Manager secret (chaos-lab/database-credentials)

Wait for: Successfully created/updated stack - chaos-lab-minimal (2-3 minutes)

What this creates:

Lambda chaos-lab-handler (5s timeout, 256MB)
Lambda chaos-lab-cpu-spike (30s timeout, 512MB)
IAM Role chaos-lab-execution-role (Bedrock + CloudWatch + Logs)
3 CloudWatch Alarms (Duration, Errors, Throttles)
1 CloudWatch Dashboard (ChaosLab-Overview)
1 Secrets Manager secret (chaos-lab/database-credentials)

Step 2: Verify Deployment

1
2
3
4
5
6
7
8
# Test Lambda responds normally

aws lambda invoke --function-name chaos-lab-handler --payload "{}" \

  --region us-east-1 --cli-binary-format raw-in-base64-out /tmp/response.json

cat /tmp/response.json



# Verify alarms are in OK state

aws cloudwatch describe-alarms --alarm-name-prefix ChaosLab \

  --query "MetricAlarms[*].[AlarmName,StateValue]" --output table --region us-east-1

Step 3: Install Python Dependencies

1
2
cd chaos-lab
pip install boto3 pyyaml rich jinja2 requests

Step 4: Run the Local Demo (No AWS Required)

1
python -m chaos_engine.cli demo

This demonstrates the full multi-agent pipeline in simulated mode without modifying any AWS resources.

Testing

Injecting Real Failures

1
2
3
4
5
6
7
cd chaos-lab/infra

# Option A: Single failure
python inject-failure.py timeout     # Lambda sleeps 60s with 5s timeout

# Option B: All failures at once
python inject-failure.py all         # Lambda + CPU + Secret + Alarm + IAM

Wait 2-3 minutes for CloudWatch alarms to fire and CloudTrail to record events.

Verifying the Failures

1
2
3
4
5
6
7
8
9
# Check alarms are firing

aws cloudwatch describe-alarms --alarm-name-prefix ChaosLab \

  --query "MetricAlarms[*].[AlarmName,StateValue]" --output table --region us-east-1



# Check CloudTrail captured the changes

aws cloudtrail lookup-events \

  --lookup-attributes AttributeKey=EventName,AttributeValue=UpdateFunctionCode20150331v2 \

  --max-results 3 --region us-east-1 \

  --query "Events[*].{Time:EventTime,Name:EventName,User:Username}" --output table

Expected output: ChaosLab-LambdaDuration in ALARM, ChaosLab-LambdaErrors in ALARM, CloudTrail showing UpdateFunctionCode, PutRolePolicy, and DeleteSecret events.

Running the Investigation

1
2
3
4
5
# Simulated mode (uses local agents, no AWS API calls)

python -m chaos_engine.cli investigate



# Or run the full demo cycle

python -m chaos_engine.cli demo

Rollback

1
python inject-failure.py rollback

This restores Lambda code, alarm thresholds, IAM policies, and secrets to their original state.

DevOps Agent Integration

The most powerful validation of this system is running the same investigation through the AWS DevOps Agent. After injecting real failures, the DevOps Agent can independently query CloudWatch, CloudTrail, and Lambda to produce its own root cause analysis which you compare against the multi-agent system's findings.

Setting Up the DevOps Agent

Create an Agent Space in the AWS DevOps Agent console
Associate your AWS account
Register Slack for notifications
Enable the MCP server power in Kiro IDE

Investigation Questions for the DevOps Agent

After injecting failures (python inject-failure.py all), ask the DevOps Agent:

Initial triage:

1
2
3
Check CloudWatch alarms with prefix "ChaosLab-", Lambda function "chaos-lab-handler" 
metrics, and CloudTrail for UpdateFunctionCode, PutRolePolicy, DeleteSecret events 
in the last 10 minutes. What's broken?

Root cause correlation:

1
2
3
I have multiple simultaneous failures: Lambda timeout, IAM AccessDeniedException on 
bedrock:InvokeModel, deleted secret, and misconfigured alarm threshold. Are these 
related? Check CloudTrail for a pattern of destructive API calls.

Recovery plan:

1
2
Give me exact AWS CLI commands to recover: fix Lambda code, restore IAM policy, 
restore secret, and fix alarm threshold. Include verification after each step.

See devops-agent-questions.md for the complete 20-question investigation guide covering all phases from detection through architecture review.

Dual Validation

Aspect	Local Multi-Agent System	AWS DevOps Agent
Data Source	Simulated signals + patterns	Real CloudWatch/CloudTrail telemetry
Speed	Instant (< 1 second)	2-10 seconds (chat) or 5-8 min (investigation)
Depth	Pattern-based findings	Live resource state analysis
Recovery	Prioritized CLI commands	Context-aware recommendations
Validation	Confidence scores	Real metric verification

When both systems agree on root cause and recovery steps, you have high confidence in the diagnosis. When they disagree, the difference reveals blind spots worth investigating.

Extending the Pattern

This architecture handles any production failure where you can define an injection method, write detection signals, and describe recovery steps. To add a new failure domain:

Add a failure to the library - Define the failure in chaos_engine/failures.py with its injection method, expected symptoms, and recovery steps.
Add an injection handler - Implement the actual AWS API calls in chaos_engine/injector.py that break the resource.
Add detection signals - Update chaos_engine/monitor.py to scan for the new failure's symptoms.
Write a runbook - Create a markdown file in runbooks/ with step-by-step recovery commands.
Update agent expertise - Adjust the relevant specialized agent's system prompt and simulated findings to cover the new domain.

For example, adding RDS failover investigation means adding a failure (rds-001: Force RDS failover), an injection handler

(rds.reboot_db_instance(ForceFailover=True)), detection signals (RDS events + connection errors), a runbook (rds_failover_recovery.md), and updating the Infrastructure Agent to check RDS metrics.

The pipeline scales horizontally: each new failure domain adds entries to the library and a runbook not new infrastructure.

Conclusion

What We Built

AgentOps Chaos Lab demonstrates that AI agents are most powerful when they work as a specialized team. No single agent can simultaneously be an expert in IAM policies, ECS container health, cost optimization, deployment pipelines, and MCP protocol compatibility. But an orchestrated group of specialists each with deep domain knowledge, sharing findings and building on each other's work can investigate complex cascading failures faster and more thoroughly than any individual approach.

The platform delivers:

16 injectable failure types across 6 categories covering real production failure modes
Multi-source signal monitoring scanning CloudWatch, CloudTrail, ECS, and Lambda simultaneously
Intelligent correlation grouping raw signals into structured incidents with confidence scores
Category-based agent dispatch ensuring the right specialists investigate each incident type
Cumulative investigation where later agents build on earlier findings
Prioritized recovery plans with exact CLI commands ordered by dependency
Dual validation comparing local agent findings against AWS DevOps Agent's real telemetry
Full lifecycle coverage from injection through detection, investigation, recovery, and prevention

The Key Insight

The AI doesn't just answer questions about metrics. It orchestrates an investigation dispatching specialists, correlating evidence from multiple AWS services, identifying cascading failure chains, and producing dependency-aware recovery plans. This is the difference between a chatbot that reads CloudWatch and an autonomous incident response system.

Cleanup

1
2
3
4
5
python infra/inject-failure.py rollback



# Delete the CloudFormation stack

aws cloudformation delete-stack --stack-name chaos-lab-minimal --region us-east-1

aws cloudformation wait stack-delete-complete --stack-name chaos-lab-minimal --region us-east-1

Built with Python, boto3, AWS CloudFormation, Lambda, CloudWatch, CloudTrail, IAM, Secrets Manager, and the Model Context Protocol. Validated against AWS DevOps Agent for real-time cloud investigation.

GitHub link: AgentOps Chaos Lab

Resources link:

AWS DevOps Agent

AWS DevOps Agent Docs

Thank you for taking the time to read my article! If you found it helpful, feel free to like, share, and drop your thoughts in the comments; I’d love to hear from you.

If you want to connect or dive deeper into cloud, AI and DevOps, feel free to follow me on my socials:

💼 LinkedIn

❌ X (formerly Twitter)

👨‍💻 DEV Community

🛡️Medium

🐙 GitHub

🌍 AWS Builder Profile

# agentic-ai # devops # kiro-agent # mcp # agents

Series: AgentOps on AWS: From Chaos to Intelligent Recovery (2 articles)

1
AWS DevOps Agent Challenge: What I Learned from Infrastructure Failures
2
AgentOps Chaos Lab: When AI Agents Become Your Incident Response Team This article

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

AgentOps Chaos Lab: When AI Agents Become Your Incident Response Team

Series: AgentOps on AWS: From Chaos to Intelligent Recovery (2 articles)

Dineshraj Dhanapathy

Table of Contents

The Problem

Solution Overview

How the Workflow Operates in Practice

Architecture

Core Concepts

Technical Deep Dive

Chaos Engine

Signal Monitoring and Correlation

Multi-Agent Orchestration

MCP Server Integration

Slack Notifications

Runbook-Driven Recovery

Deployment Steps

Step 2: Verify Deployment

Step 3: Install Python Dependencies

Step 4: Run the Local Demo (No AWS Required)

Testing

Injecting Real Failures

Running the Investigation

Rollback

DevOps Agent Integration

Setting Up the DevOps Agent

Investigation Questions for the DevOps Agent

Dual Validation

Extending the Pattern

Conclusion

What We Built

The Key Insight

Cleanup

Series: AgentOps on AWS: From Chaos to Intelligent Recovery (2 articles)