Behind the Scenes: What Happens When an EC2 Instance Becomes Unhealthy
Overview of what happens when an EC2 instance is launched, from compute allocation and virtualization to storage, networking, and OS initialization.
Series: Behind the Scenes (3 articles)
- 2Behind the Scenes: What Happens When an EC2 Instance Becomes Unhealthy This article
What Happens When an EC2 Instance Becomes Unhealthy?
In a highly available AWS environment, an EC2 instance is not expected to run forever. Instances can become unhealthy because of application failures, operating system issues, network problems, or underlying infrastructure failures.
The important part is not preventing every failure. It is making sure that a single unhealthy instance does not become a service outage.
But what actually happens when an EC2 instance becomes unhealthy?
Instance State vs. Instance Health
One important distinction is the difference between instance state and instance health.
An EC2 instance can be in the
running state while the application running on it is not responding.For example, the operating system may still be running, but:
- The application process may have stopped.
- The server may not respond to requests.
- The application may be returning errors.
- The instance may have a network-related problem.
This is why simply checking whether an instance is running is not enough for a highly available application.
Health Checks and Failure Detection
When EC2 instances are placed behind an Elastic Load Balancer, health checks can be used to determine whether an instance is capable of receiving traffic.
A health check can test a specific endpoint, port, or protocol.
For example:
Load Balancer → HTTP request → EC2:8080/health
If the expected response is not received according to the configured health check rules, the target can be marked as unhealthy.
The Load Balancer then stops routing new traffic to that target.
This is the first important part of the recovery process:
Detect the failure → Stop sending traffic
Handling Existing Traffic
Once a target becomes unhealthy, the Load Balancer avoids sending new requests to it.
The exact behavior of existing connections depends on the protocol, connection state, and Load Balancer configuration. This is why health checks should be designed around the actual application behavior rather than simply checking whether a port is open.
A health endpoint can also be more useful than a basic connectivity check.
For example, an application might be able to answer on port 8080 even though its database connection is unavailable.
A simple TCP check could consider the instance healthy, while an application-level health check could identify the real problem.
Auto Scaling and Instance Replacement
A Load Balancer can stop sending traffic to an unhealthy instance, but this does not automatically mean that the instance will be replaced.
This is where an Auto Scaling Group (ASG) becomes important.
An Auto Scaling Group maintains the desired capacity of an application environment.
For example:
Desired capacity: 3
If one instance becomes unhealthy:
3 instances → 2 healthy instances → unhealthy instance replaced → 3 instances
The replacement instance is launched according to the configuration defined by the associated Launch Template.
This means the recovery process can happen without manually logging into a server and rebuilding it.
The Recovery Flow
A simplified recovery flow looks like this:
EC2 Instance
↓
Health Check Fails
↓
Load Balancer Stops Routing Traffic
↓
Auto Scaling Detects Unhealthy Capacity
↓
Unhealthy Instance Is Replaced
↓
New EC2 Instance Is Launched
↓
New Instance Passes Health Checks
↓
Traffic Is Sent to the New Instance
↓
Health Check Fails
↓
Load Balancer Stops Routing Traffic
↓
Auto Scaling Detects Unhealthy Capacity
↓
Unhealthy Instance Is Replaced
↓
New EC2 Instance Is Launched
↓
New Instance Passes Health Checks
↓
Traffic Is Sent to the New Instance
This is one of the key ideas behind highly available cloud architectures: failure becomes part of the normal operating model.
Data and Persistent Storage
This architecture also raises an important question:
What happens to the data stored on the failed instance?
If an application stores important state only on the local instance, replacing that instance can cause problems.
For this reason, scalable applications generally separate compute from persistent data.
Depending on the workload, data might be stored in services such as:
- Amazon RDS
- Amazon S3
- Amazon EBS
- ElastiCache
The goal is to make the EC2 instance replaceable without losing important application state.
Underlying Infrastructure Failures
Not every failure is caused by the application.
AWS also provides mechanisms for detecting certain underlying infrastructure problems and recovering affected instances.
For example, EC2 Auto Recovery can automatically recover an instance when certain underlying hardware or system failures are detected, depending on the instance and configuration.
This is different from an Auto Scaling replacement.
Auto Scaling is generally concerned with maintaining application capacity, while EC2 recovery mechanisms can address failures affecting the underlying instance infrastructure.
Designing for Replaceable Compute
The important concept is that a production EC2 instance should not necessarily be treated as a permanent server.
Instead, it can be treated as a replaceable compute resource.
This changes the architecture from:
One server → Manual troubleshooting → Manual recovery
to:
Multiple instances → Health checks → Automated traffic management → Automated replacement
This approach reduces the impact of individual instance failures and allows the infrastructure to recover without making every failure a manual operational task.
As a summary, an unhealthy EC2 instance does not necessarily mean that the entire application is unavailable.
With the right architecture, the failure can trigger a sequence of automated actions:
Detect → Isolate → Replace → Validate → Restore traffic
Load Balancers, Auto Scaling Groups, Launch Templates, and persistent data services each play a different role in this process.
The key idea is simple:
In a resilient cloud architecture, an unhealthy instance is a condition to handle—not necessarily an incident that requires manual recovery.
Series: Behind the Scenes (3 articles)
- 2Behind the Scenes: What Happens When an EC2 Instance Becomes Unhealthy This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article