Multi-Agent Chaos Engineering
This article describes a multi-agent solution for chaos engineering built using the Strands SDK.
We've observed a common pattern across organizations: testing is often deprioritized in the software development lifecycle, despite being fundamental to system resilience. This oversight is particularly problematic when you consider that most production outages stem from misconfigurations and bugs—the very issues comprehensive testing is designed to catch.
The reasons are predictable yet concerning. Competing priorities, constrained resources, and tight budgets push testing to the sidelines in favor of new features and faster time-to-market. In today's competitive landscape and unpredictable economy, the pressure to deliver quickly often overshadows the need to test. This creates a costly cycle where skipped testing leads to production issues, outages, and emergency fixes that consume far more time and resources than the original testing would have required. Organizations can end up spending more time firefighting than building new capabilities, while teams become frustrated and disengaged when constant crisis management prevents them from focusing on meaningful innovation.
As organizations build increasingly complex, distributed systems, incorporating chaos engineering and fault experimentation into their development cycle becomes more important. With business impacts from outages growing more severe and users demanding higher uptime and performance, understanding system behavior under stress becomes increasingly valuable. This proactive approach—deliberately introducing controlled failures—helps teams understand how their systems behave under various failure modes before those failures occur naturally in production.
While chaos engineering is straightforward in concept, many organizations perceive the barrier to entry as higher than it actually is. The apparent complexity of hypothesis generation, experiment design, and stakeholder coordination often prevents organizations hesitant to invest time and resources from getting started. Just like any other part of the development lifecycle, we think agentic assistants can help automate this process and make chaos engineering more approachable. This is where generative AI becomes a game changer, automating the chaos engineering lifecycle—from generating failure hypotheses based on source code analysis and ranking them by business impact, to creating AWS FIS experiment templates, executing experiments, and facilitating continuous learning and improvement. By dramatically lowering these perceived barriers, AI makes chaos engineering accessible to organizations at any maturity level, representing a transformative approach to building resilience practices while maintaining the reliability their customers expect.
Why agents
We choose to use a multi-agent solution to solve this problem as it felt like the most flexible approach. We knew what the major steps were going to be at each milestone, but the details could change depending on the workload. By leveraging agents that had access to tools we could guide them to dynamically discover the resources and create the experiments. Having agents with specific tasks also allows for integrating new tools in the future - for example, using Gremlin rather than the AWS FIS service.
We also saw chaos engineering agents as a gap in the agentic software development lifecycle. Using generative AI to write code is table stakes at this point. Where we’re starting to see more mature developers go is purpose built agents collaborating together. For example a backend agent could hand off a design spec to a frontend specialist. These artifacts would then get passed to a documentation or QA agent for further updates. Using this toolkit we can give developers the visibility to design with resiliency in mind from the beginning. We could also potentially expand our solution to give developers near realtime feedback on how changes could impact their applications resiliency posture.
Multi-agent design
The current solution was built using the Strands SDK with purpose-built agents for each function. Each agent has its own prompt, model configuration, set of tools, and even sometimes sub agents. Using these different agents allowed us to create clear context and responsibility boundaries. It also accelerated our development efforts as we could have different team members working in their own domains without affecting each other.
Hypothesis generator agent
The first step in the process is generating hypotheses—theories about the workload’s resilience—and this process is orchestrated by the hypothesis generator agent. It coordinates multiple specialized sub-agents: the source code analysis agent examines a source-code repository to analyze infrastructure-as-code and application code patterns, the AWS resource analysis agent discovers and analyzes deployed AWS resources, and the hypothesis builder agent generates the actual hypotheses based on the combined analysis. Its primary responsibility is to ensure the chaos tests will provide comprehensive coverage of the workload. It does this by producing 20-100 hypotheses across different failure domains.
Hypothesis prioritization agent
This agent retrieves all hypotheses, compares them with its prioritization criteria (considering factors like business impact, safety, learning value, and implementation complexity), and then generates a ranked priority list. The agent has access to database tools for efficient bulk updates, plus file system tools for handling complex analysis when dealing with large hypothesis sets (50+ hypotheses).
Experiment design agent
The experiment design agent is an AWS FIS expert that converts prioritized hypotheses into production-ready FIS experiment templates. It retrieves the corresponding hypotheses from the database and validates them against AWS FIS capabilities using live documentation from an Amazon Bedrock KnowledgeBase. It performs validation including FIS compatibility checks and resource targeting before saving experiments to the database, handling both compatible experiments (saved as "draft") and incompatible ones (saved as "validation_failed" with detailed notes).
Experiments agent
The experiments agent handles the complete lifecycle of FIS experiments after they’re designed. The agent has access to the built-in Strands use_aws tool for all AWS API operations (resource discovery, FIS experiment creation, and execution). It starts by reviewing the resource discovery previously done, and checking AWS resources that match experiment targets (validating clusters, namespaces, pod selectors for EKS or task definitions for ECS). Next it creates FIS experiment templates with validated resource ARNs, updating the database status at each step (validating → creating → created → running → completed). Finally the agent can execute experiments while monitoring progress and capturing detailed results or failure information in the database.
Learning and iteration agent
The learning and iteration agent analyzes the completed chaos engineering experiments to extract insights and drive continuous improvement in system resilience practices. The agent synthesizes learnings from multiple experiments to identify root causes of system weaknesses, correlate failures with system conditions, generate statistical insights from outcomes, and provide strategic guidance on resilience investments while maintaining a comprehensive learning history for the organization.
Hypothesis evaluator agent
The hypothesis evaluator agent provides objective quality assessment of chaos engineering hypotheses using an LLM-as-judge evaluation pattern, scoring hypotheses based on how well they can be implemented with AWS FIS, how precisely they define failure conditions, how realistic the scenarios are, how safely bounded the blast radius is, and how much learning value they provide. It then stores all evaluation data in the database and returns summary statistics including score distributions and individual evaluation rationales.
Example
For our example, we used the AWS Retail Store Sample container app. This is a good example as it contains different microservices with various datastores we can use for testing.
Hypothesis generation
The first step is to run the hypothesis generation agent. This agent creates about 40 hypotheses.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
A total of 40 chaos engineering hypotheses have been successfully generated across multiple failure domains:
1. **Compute Failures (9 hypotheses)**:
- ECS task termination impacts
- Resource exhaustion scenarios
- Multiple service failures
- CPU and memory pressure situations
2. **Data Layer Failures (11 hypotheses)**:
- PostgreSQL primary instance failures
- Connection pool exhaustion
- High CPU/memory load on databases
- Replication lag scenarios
- Database storage issues
- DocumentDB instance failures
- Redis cache failures
3. **Network Failures (10 hypotheses)**:
- Service-to-service communication issues
- Network latency and jitter
- Network partitions
- Database connectivity problems
- API gateway interruptions
- Cross-VPC communication failures
- Service discovery issues
4. **Security Failures (5 hypotheses)**:
- IAM role permission restrictions
- Security group rule removal
- SSL certificate expiration
- Secret rotation failures
- CORS misconfiguration
5. **Configuration Failures (5 hypotheses)**:
- Environment variable misconfigurations
- Database connection string corruption
- Service discovery misconfiguration
- API rate limit configuration issues
- Container resource limit misconfigurations
6. **Resource Constraints (7 hypotheses)**:
- Memory exhaustion
- CPU throttling
- Database disk space issues
- Cache memory fragmentation
- Network bandwidth constraints
- Container launch rate limiting
- Connection pool exhaustion
7. **Monitoring Failures (3 hypotheses)**:
- CloudWatch metrics delays
- OpenTelemetry collector failures
- CloudWatch alarm failuresHypothesis prioritization
The next agent prioritizes the available hypotheses.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
I organized the 50 hypotheses into five tiers:
### Tier 1 (Priorities 1-10): Critical Business Function Failures
These tests focus on core revenue-generating paths and fundamental infrastructure with controlled risk, including checkout flows, database resilience, and core services.
### Tier 2 (Priorities 11-20): Key Supporting Services
Tests that target important supporting services with moderate business impact, including cache services, ordering systems, and network partitions.
### Tier 3 (Priorities 21-30): Infrastructure Resilience
Tests that validate underlying infrastructure resilience with broader impact, including cluster resource exhaustion, service discovery, and replication.
### Tier 4 (Priorities 31-40): Edge Cases and Configuration Failures
Tests that validate system behavior during unexpected configuration changes, including security group changes, DNS failures, and certificate issues.
### Tier 5 (Priorities 41-50): Advanced Scenarios
Complex scenarios that test multiple failure modes or have larger blast radius, including monitoring system failures, telemetry issues, and complex configuration failures.
## Top 10 Priority Experiments
1. **Checkout Service Dependency Failure (#16)** - Directly impacts revenue conversion; critical path for business
2. **PostgreSQL Primary Instance Failure (#6)** - Core database resilience; fundamental to all operations
3. **Cart Service CPU Throttling (#2)** - Key customer touchpoint; affects purchase completion
4. **ECS Task Termination Impact (#1)** - Foundational compute resilience; affects all services
5. **API Gateway Connection Interruption (#23)** - All external API access; high customer visibility
6. **Redis Cache Failure (#12)** - Key performance component; affects entire application
7. **PostgreSQL Connection Pool Exhaustion (#7)** - Common failure pattern; high learning value
8. **DocumentDB Instance Failure (#9)** - Document store resilience; affects product data
9. **UI Service Network Latency (#5)** - Direct customer experience impact; frontend resilience
10. **Security Group Rule Restriction (#27)** - Service isolation testing; high security relevanceConverting hypotheses to experiments
The next agent creates FIS experiments for each hypothesis. Here’s an example FIS experiment definition that the agent created.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
{
"description": "Test application handling of PostgreSQL read replica lag",
"targets": {
"rdsInstance": {
"resourceType": "aws:rds:db",
"selectionMode": "COUNT(1)",
"parameters": {
"dbType": "INSTANCE",
"instanceID": "user-database"
}
}
},
"actions": {
"stressCPU": {
"actionId": "aws:rds:stress-instance",
"parameters": {
"duration": "PT15M",
"stressType": "cpu",
"percentage": "80"
},
"targets": {
"dbInstances": "rdsInstance"
}
}
},
"stopConditions": [
{
"source": "none"
}
],
"roleArn": "arn:aws:iam::XXXXXXXXXX:role/ChaosAgentFISExecutionRole",
"tags": {
"Name": "PostgreSQL-Read-Replica-Lag"
}
}Experiment execution
The next agent executes the experiments. Here’s a sample result table - we’ve found a few problems!
1
2
3
4
5
6
7
8
9
## Experiment Results
| # | Experiment Title | Status | Duration | Notes |
|---|------------------|--------|----------|-------|
| 1 | Cart Service IAM Access Blocked Test | Failed | 2 seconds | Authorization failure when accessing EKS pods |
| 2 | Multiple ECS Service Failures Test | Failed | 2 seconds | Authorization failure when accessing EKS recommendation pods |
| 3 | Catalog to PostgreSQL Packet Loss Test | Failed | 55 seconds | DependentService Failure for action inducePacketLoss. Unable to heartbeat FIS POD |
| 4 | PostgreSQL Read Replica Lag Test | Completed | 13 seconds | Successfully rebooted PostgreSQL database instance |
| 5 | Catalog Service Memory Exhaustion Test | Failed | 53 second | DependentService Failure for action exhaustMemory. Unable to heartbeat FIS POD |Result analysis
The next agent analyzes the experiment results and provides recommendations for improvement. Here’s a snippet of the output.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
## Key Insights and Learnings
1. The application architecture has properly implemented database read/write splitting, but lacks similar resilience patterns in other areas.
2. Several critical resilience gaps were identified:
- Inadequate error handling for authentication failures
- Missing circuit breakers and fallback mechanisms
- Insufficient connection retry logic
- Improper memory management
3. The system architecture appears to have tighter coupling between services than anticipated, leading to cascading failures.
4. Connection pool and memory management configurations are not optimized for resilience.
5. The organization would benefit from a more comprehensive chaos engineering practice with regular testing.Evaluation
We’ve also included an agent that evaluates the generated hypotheses along the dimensions of testability, specificity, realism, safety, and learning value. Here’s how it ranked five of the hypotheses.
Conclusion
This multi-agent project tries to partially automate the chaos engineering process for a software project. This project is only for demonstration purposes. Before using it in production, you should evaluate it in a non-production setting, customize it to your needs, and perform a risk mitigation assessment. You can view the code at https://github.com/aws-samples/sample-strands-chaos-engineering-agents/tree/main .
We also hope this example shows a realistic application of multi-agent systems, with multiple specialized agents coming together to perform a specific task.
Credits
Our team included David Christian, Jennifer Moran, Robert Fisher, and Swagat Kulkarni.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article