What If AWS Failed Tomorrow? Put Your Architecture to the Test with AWS FIS
Would you trust a system that has never been tested under adverse conditions? That was the question I opened my talk with at AWS Community Day Bolivia 2026, in Santa Cruz. Most teams design for failure: Multi-AZ, Auto Scaling, health checks, backups. Very few deliberately test it.
A system that has never failed may not be resilient
Would you trust a system that has never been tested under adverse conditions? That was the question I opened my talk with at AWS Community Day Bolivia 2026, in Santa Cruz. Most teams design for failure: Multi-AZ, Auto Scaling, health checks, backups. Very few deliberately test it.
Werner Vogels summed it up years ago: "Everything fails, all the time." Networks fail, components fail, dependencies fail, and applications fail. The goal is not to eliminate failure; it is to be prepared for it. In this article I share how to put that idea into practice with Chaos Engineering and AWS Fault Injection Service (FIS).
"It's in the cloud, so it won't go down"
This is one of the most dangerous myths in our industry. Cloud does not equal automatic high availability: high availability is the result of architecture plus decisions.
- The cloud provides Multi-AZ, Auto Scaling, backups, and observability.
- You must design redundancy, failover, recovery, and limits.
- And you must test that it actually works when something breaks.
The October 20, 2025 event in us-east-1 made this clear. A DNS resolution issue related to DynamoDB propagated to dependent services such as EC2, NLB, Lambda, and CloudWatch, and from there to applications around the world. The lesson: dependencies amplify incidents.
AWS will recover. The uncomfortable question is a different one: will your application recover? And how do you know if you have never tested it?
It helps to separate three concepts we often mix up:
| Concept | Question it answers |
|---|---|
| Availability | Is it available right now? |
| Reliability | Does it work correctly for the expected time? |
| Resilience | Does it keep operating, or recover, when something fails? |
Designing for failure is not the same as testing for failure.
What Chaos Engineering is (and isn't)
Chaos Engineering means experimenting deliberately to uncover weaknesses before a real incident does. It is not randomly breaking production, shutting down servers for fun, or creating uncontrolled incidents. Nor is it a "let's see what happens."
Every experiment follows the scientific method:
- Steady state: define what "normal" means.
- Hypothesis: for example, "if we lose an EC2 instance, the application will keep responding correctly."
- Inject the fault.
- Observe the behavior.
- Compare against the steady state.
- Learn and improve.
The steady state is expressed through business and user metrics, not infrastructure metrics. For example: latency < 300 ms, error rate < 1%, availability > 99.9%, and the expected throughput. The question is not "did EC2 go down?" but "did the user experience change?"
The five principles of Chaos Engineering sum it up: build a hypothesis around steady state, simulate real-world events (latency, loss, outages, saturation), experiment under real conditions, automate continuously, and minimize the blast radius.
That last point is key. Start small, learn, and expand: one instance, then 10% of the fleet, then an Availability Zone, then dependencies and, only with maturity, a Region. The goal is maximum learning with the lowest possible risk.
From Chaos Monkey to AWS FIS
Between 2008 and 2010, during its migration to AWS, Netflix realized that instances would disappear sooner or later. That is how Chaos Monkey was born (2010–2011): a tool that randomly terminated instances behind a target group to answer a single question. If EC2-B disappears, does the application keep working for the user?
Then came the Simian Army (2011–2012), the Principles of Chaos Engineering (2015) and, in 2021, AWS Fault Injection Service. Today experiments reach Availability Zone and Region scenarios. We went from "waiting for failure" to "experimenting with it."
AWS FIS is a managed service for running fault injection experiments in a controlled way. Each experiment is defined in a repeatable experiment template with five pieces:
| Piece | Question it answers |
|---|---|
| Targets | Which resources will I affect? |
| Actions | Which fault do I introduce? |
| Duration | How long does it last? |
| Stop conditions | When do I abort? |
| Observability | What happened and what did I learn? |
FIS can disrupt compute (EC2, Auto Scaling: stop, reboot, capacity errors), containers (EKS, ECS: CPU, memory, latency, pod disruption), network (latency, packet loss, connectivity in the VPC), and data (RDS failover, EBS I/O latency). Targets are selected by ID or by tags, which lets you control scope precisely.
Stop conditions are the red button. They are tied to CloudWatch alarms: if the error rate exceeds, for example, 5%, FIS stops the experiment automatically.
Let's design our first experiment
The experiment I showed in the demo is deliberately simple: stop one EC2 instance for 3 minutes behind an Application Load Balancer with instances spread across multiple AZs.
- Hypothesis: if we lose one instance, the ALB reroutes traffic and the user doesn't notice.
- Target: 1 EC2 instance tagged
chaos-ready=true. - Action:
aws:ec2:stop-instances, restarting automatically after 3 minutes. - Stop condition: a CloudWatch alarm on the ALB error rate.
- Observation: response time and availability.
The JSON template (replace the values between
<>):1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
{
"description": "Stop 1 EC2 instance for 3 minutes",
"targets": {
"one-instance": {
"resourceType": "aws:ec2:instance",
"resourceTags": { "chaos-ready": "true" },
"selectionMode": "COUNT(1)"
}
},
"actions": {
"stop-ec2": {
"actionId": "aws:ec2:stop-instances",
"parameters": { "startInstancesAfterDuration": "PT3M" },
"targets": { "Instances": "one-instance" }
}
},
"stopConditions": [
{
"source": "aws:cloudwatch:alarm",
"value": "arn:aws:cloudwatch:<REGION>:<ACCOUNT_ID>:alarm:<ALARM_NAME>"
}
],
"roleArn": "arn:aws:iam::<ACCOUNT_ID>:role/<FIS_ROLE>",
"tags": { "Name": "chaos-ec2-stop" }
}Create and run the experiment:
1
2
3
aws fis create-experiment-template --cli-input-json file://template.json
aws fis start-experiment --experiment-template-id <TEMPLATE_ID>
aws fis get-experiment --id <EXPERIMENT_ID>While it runs, watch your CloudWatch dashboard. If the hypothesis holds, you have evidence that your Multi-AZ redundancy and health checks work. If not, you found a single point of failure in a controlled experiment rather than at 3 a.m.
Building a culture of resilience
The real goal is not to break things; it is to turn resilience into evidence. Hypothesis, experiment, failure, observation, learning, and improvement form a cycle that repeats, not a one-time event.
Three recommendations to get started:
- Start in a non-production environment with a minimal blast radius and stop conditions always configured.
- Define the steady state with user metrics before injecting any fault.
- Automate your experiments in your pipelines so resilience is validated with every change, not once a year.
We can't control when the next incident will happen, but we can control when we put our architecture to the test. Resilience isn't about avoiding every failure; it's about being ready when they happen.
So, do you already know what would happen if AWS failed tomorrow?
It was a truly enriching experience, with a nearly full auditorium and great energy from everyone who attended. I especially want to thank those who joined me for my talk, and also for their understanding and patience with the issues I had with my voice that day. Despite that, it was an experience I enjoyed immensely, and it reaffirmed the value of sharing, learning, and growing together as a community.
A Special Thanks to AWS New Voices
I would also like to express my sincere gratitude to the AWS New Voices program for the valuable training, guidance, and support throughout this journey. The skills and confidence I gained from the program were incredibly helpful in preparing and delivering my talk at AWS Community Day Bolivia 2026.
Thank you for empowering community speakers to share their knowledge, tell their stories, and inspire others through technology!
Oscar Murguia — DevOps Engineer CoE at Banco Solidario S.A., AWS Community Builder, and AWS User Group Leader. Based on the talk presented at AWS Community Day Bolivia 2026 (Santa Cruz, August 29, 2026).
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article