
The EC2 Instance Was Healthy. The Application Wasn't — Here's How I Found the Problem
A real-world AWS production troubleshooting story about an EC2 instance that appeared healthy while the application behind an Application Load Balancer was failing. Covers ALB 504 errors, Apache, PHP-FPM, application health checks, and a request-path approach to troubleshooting.
A few months ago, I was troubleshooting an application outage where the first thing that confused me was this:
The EC2 instance looked completely healthy.
CPU was normal.
Memory wasn't under pressure.
EBS looked fine.
EC2 status checks were passing.
But the application behind the load balancer was still failing.
That was a good reminder that:
EC2 health and application health are two completely different things.
What I Saw
The setup was roughly:
Client → ALB → EC2 → Apache → PHP-FPM → Application
There were two EC2 instances behind the Application Load Balancer.
One instance was working normally.
The other one was causing requests to fail.
At first glance, there wasn't anything obviously wrong with the instance.
I checked the usual things.
topCPU looked normal.
Then:
free -mMemory wasn't the issue.
I checked disk:
df -hNothing alarming there either.
EC2 status checks were also passing.
So the obvious question was:
What exactly is failing?
I Stopped Looking at EC2 and Started Looking at the Application
This was the important change in my troubleshooting approach.
Instead of asking:
"Is the EC2 instance healthy?"
I started asking:
"Can this instance actually serve the application request?"
The ALB was giving me a much better signal.
One target was healthy.
The other wasn't.
That immediately narrowed the problem down to the application path on that particular instance.
At that point, I stopped treating this as an EC2 health problem and started treating it as an application request-path problem.
Checking the Web Server
I checked Apache:
systemctl status httpdThen I went through the application logs.
Eventually, I found something interesting around PHP-FPM.
There were a large number of PHP-FPM worker processes.
The server wasn't necessarily CPU-bound.
The problem was that the application workers were getting consumed.
That explained why the instance could look "healthy" from an infrastructure perspective while the application was struggling.
This was an important distinction:
Infrastructure looks healthy → Application is still unhealthy
This Is Where Monitoring Can Fool You
If your monitoring only looks at:
- CPU
- Memory
- Disk
- EC2 status checks
you can easily miss an application-level failure.
Imagine:
- CPU: 5% ✓
- Memory: 40% ✓
- Disk: Normal ✓
- EC2 Status: OK ✓
- Application: Failing ✗
Everything looks green.
Except the thing your customer actually cares about:
The application.
That's why infrastructure metrics alone aren't enough for production workloads.
What I Changed
I worked through the PHP-FPM configuration and adjusted the worker settings based on the workload instead of blindly increasing values.
I also added a simple health endpoint:
/health.html The purpose was simple.
I wanted the load balancer to be able to answer:
"Can this server actually respond to an application health request?"
rather than relying only on whether the EC2 instance itself was alive.
A health endpoint is a small thing, but it gives you a much better signal about the actual application path.
Then I Looked at the ALB
The ALB logs were particularly useful.
There were a number of
504 responses.That changed the investigation again.
A
504 from the ALB doesn't automatically mean:"The ALB is broken."
It can mean that the ALB didn't receive a response from the target within the expected time.
So I started looking at the complete request path:
Client → ALB → EC2 → Apache → PHP-FPM → Application
The question became:
Where exactly is the request getting stuck?
That question was much more useful than simply asking whether the EC2 instance was healthy.
One Thing I Learned From This Incident
I used to think about infrastructure troubleshooting mainly in terms of:
- CPU
- Memory
- Disk
- Network
Those are still important.
But production troubleshooting taught me that you also need to think in terms of the request path.
For example:
- Is the ALB receiving the request?
- Is it reaching the target?
- Is Apache responding?
- Is PHP-FPM accepting the request?
- Is the application processing the request?
- Is the backend or database responding?
Instead of checking random metrics, I can now ask:
Where does the request stop?
That mental model is much more useful.
What I Would Check Today
If I saw the same issue again, my first checks would be:
1. ALB Target Health
First:
Which target is unhealthy?
If one target is healthy and another isn't, that's already a valuable clue.
2. ALB Access Logs
I would check the response codes and timing.
For example:
504A 504 tells me that I need to investigate the communication between the load balancer and the target rather than immediately blaming the load balancer.
3. EC2
I would still check the basic system health:
topfree -mdf -hI don't skip these checks.
I just don't stop there.
4. Web Server
For Apache:
systemctl status httpdThen check the relevant Apache error logs.
I'm looking for things like:
- connection problems
- request failures
- worker issues
- upstream errors
- configuration problems
5. PHP-FPM
Check the service:
systemctl status php-fpmThen inspect the running processes:
ps aux | grep php-fpmDepending on the application, I would also investigate:
- worker utilization
-
max_children
- slow requests
- process limits
- PHP-FPM logs
6. Application Logs
This is where the application itself becomes the source of truth.
Look for:
- timeouts
- worker exhaustion
- slow requests
- application exceptions
- database connectivity problems
- external dependency failures
The infrastructure can look completely normal while the application is throwing errors continuously.
7. Test Locally
Instead of testing only through the ALB, test directly from the instance.
For example:
curl -I http://localhost/Or test the health endpoint:
curl -i http://localhost/health.htmlThis is useful because it helps separate:
ALB problem
from:
Application / EC2 problem
If the application fails locally on the instance, there is no point spending the next hour changing ALB settings.
The Part I Wouldn't Automate Immediately
It's tempting to build something like:
Target unhealthy → Restart EC2
I wouldn't do that immediately.
You can make an incident worse by automatically restarting an instance before collecting evidence.
You might lose valuable information about what was actually happening.
I'd rather have:
- Target becomes unhealthy
- Collect diagnostics
- Check the application
- Attempt safe remediation
- Validate
- Escalate if still broken
Automation should help engineers troubleshoot.
It shouldn't just hide the evidence.
Another Lesson: Replacing the Instance Isn't Always Enough
During incidents, it's easy to think:
"The instance is bad. Replace it."
Sometimes that works.
But what if the problem is actually in:
- the AMI
- Launch Template
- application configuration
- PHP-FPM configuration
- environment variables
- deployment
- dependency configuration
Then the replacement instance can simply recreate the same problem.
That's why configuration consistency matters.
If I make a manual fix on one EC2 instance, I also need to ask:
"Would a newly launched instance have this same configuration?"
If the answer is no, the fix isn't finished.
My Biggest Takeaway
The biggest lesson from this incident wasn't a specific Apache command or PHP-FPM setting.
It was this:
Don't troubleshoot the server. Troubleshoot the request.
When a customer says:
"The website is down."
Don't start with:
"Is EC2 running?"
Start thinking about the complete request path:
Customer → DNS → CloudFront / ALB → Target → Web Server → Application Runtime → Database / Dependencies → Response
Then ask:
Where does the request stop?
That's usually where the real problem is.
What I Would Monitor Now
For this type of application, I'd want visibility into more than CPU and memory.
I'd monitor:
- ALB healthy/unhealthy targets
- ALB 4xx/5xx responses
- Target response time
- Application health endpoint
- Web server errors
- PHP-FPM worker utilization
- Application errors
- Database connectivity
- Dependency latency
Because:
A server can be healthy while your application is completely broken.
And that's a distinction that becomes very important once you're dealing with production systems.
What This Changed in My Troubleshooting Approach
Before this incident, it was easy to follow a checklist like:
CPU?Memory?Disk?EC2 status?Now I think more like this:
- Customer reports an issue
- Identify the exact symptom
- Find where the request fails
- Collect evidence
- Make the smallest safe change
- Validate the fix
- Prevent recurrence
That approach has saved me from making a lot of unnecessary changes during production troubleshooting.
Final Thought
One of the biggest mistakes in cloud operations is assuming:
"The server is healthy, so the application must be healthy."
That's simply not true.
Infrastructure health is one layer.
Application health is another.
And customer experience is another.
The job of a cloud engineer is to connect those layers and find where the actual failure is happening.
I'm documenting more of these real-world AWS troubleshooting scenarios in my AWS Production Troubleshooting repository.
The next one will probably be another incident where the obvious AWS metric wasn't actually the problem.
Have you ever had an EC2 instance showing completely normal metrics while the application was still down? What did you find?
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article