AWS Builder Center

Builder Center

Your AWS Application Is Running ... But Can You Explain When It Breaks?

Deploying an application to AWS is only half the job. This article explores how to design applications with observability in mind, connect logs, metrics, traces, and deployments, and systematically move from a user-reported problem to its actual root cause.

Getting an application running on AWS can feel like the finish line.
You create your infrastructure, deploy your application, configure DNS, connect your database, and everything works.
Then one day:
Users start getting 500 errors.
Or:
The application suddenly becomes slow.
Or:
Your database is running, but requests are timing out.
At that point, deploying the application isn't the difficult part anymore.
Finding out what went wrong is.
This is where observability becomes one of the most important parts of cloud architecture.

A Production Application Is a Chain of Dependencies

Consider a relatively simple AWS architecture:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
User
|
v
Route 53
|
v
CloudFront
|
v
ALB
/ \
v v
EC2 EC2
\ /
\ /
RDS
There are multiple places where a request can fail.
The user may see:
1
HTTP 500 Internal Server Error
But that doesn't tell you where the problem actually occurred.
The problem could be:
  • DNS configuration
  • CloudFront
  • Load balancer
  • EC2
  • Application code
  • Database connectivity
  • Database performance
  • Security groups
  • Network routing
  • An external API
The visible symptom and the actual root cause can be completely different.

Don't Start Debugging at the Server

One of the most common mistakes is immediately connecting to the EC2 instance.
Instead, follow the request through the architecture.
For example:
1
2
3
4
5
6
7
8
9
10
11
12
13
User
↓
DNS
↓
CloudFront
↓
ALB
↓
EC2
↓
Application
↓
RDS
At every stage, ask:
Did the request successfully reach this component?
This gives you a structured debugging process instead of randomly changing configurations.

Logs Tell You What Happened

Logs are one of the most useful sources of information during an incident.
A useful application log might look like:
1
2
3
4
5
12:41:03 GET /api/books/123
12:41:03 Authentication successful
12:41:03 Database query started
12:41:04 Database query completed
12:41:04 Response 200
Now imagine:
1
2
3
4
5
12:41:03 GET /api/books/123
12:41:03 Authentication successful
12:41:03 Database query started
12:41:33 Database timeout
12:41:33 Response 500
The user only sees:
1
500 Internal Server Error
But the logs tell us something much more useful:
The request reached the application, but the database operation timed out.
That immediately narrows the investigation.

Metrics Tell You What Changed

Logs are useful for individual events.
Metrics help you understand the bigger picture.
Imagine your CloudWatch dashboard shows:
1
2
3
4
5
6
12:00 Normal traffic
12:05 CPU begins increasing
12:07 Database connections increase
12:09 Request latency increases
12:10 HTTP 500 errors increase
12:12 Users report failures
Now we have a timeline.
Instead of asking:
"Why is the application broken?"
We can ask:
"What changed around 12:05?"
That is a much better debugging question.

The Importance of Correlation

The real power comes from combining different sources of information.
Imagine:
1
2
3
4
5
6
7
8
9
Deployment
↓
CPU increase
↓
Database connections increase
↓
Latency increases
↓
5xx errors increase
Individually, each event might not mean much.
Together, they tell a story.
This is why production observability should connect:
  • Logs
  • Metrics
  • Traces
  • Deployments
  • Infrastructure events
  • Application errors
  • Alerts
The goal isn't simply to collect more data.
The goal is to connect the data.

A Deployment Should Be Part of Your Investigation

Suppose your application was working perfectly for six hours.
Then you deploy version v2.4.1.
Five minutes later:
1
2
3
Error rate: ↑
Latency: ↑
Database load: ↑
The deployment is now an important event in the investigation.
You should be able to look at your monitoring system and immediately see:
1
2
3
4
5
6
7
8
9
12:00 — Application healthy

12:15 — Deployment v2.4.1

12:19 — Latency increased

12:21 — 5xx errors increased

12:23 — Alarm triggered
This doesn't automatically prove that the deployment caused the problem.
But it gives you a strong starting point for investigation.

Think in Questions, Not Dashboards

A dashboard containing 50 graphs isn't necessarily useful.
A better dashboard answers important questions.

Is the application healthy?

Look at:
  • Request rate
  • Error rate
  • Latency

Are the servers healthy?

Look at:
  • CPU
  • Memory
  • Disk
  • Network
  • Instance health

Is the database healthy?

Look at:
  • Connections
  • CPU
  • Storage
  • Latency
  • Query performance

Is the infrastructure changing?

Look at:
  • Deployments
  • Scaling events
  • Configuration changes
  • Instance replacements
Good observability isn't about having the most graphs.
It's about having the right information when something goes wrong.

Trace the Request

For distributed applications, logs and metrics may not be enough.
Imagine:
1
2
3
4
5
6
7
8
9
10
11
API
↓
Authentication Service
↓
Book Service
↓
Recommendation Service
↓
Database
↓
External API
A single user request might touch several components.
If the request takes 8 seconds, which service caused the delay?
Distributed tracing can help answer that.
Conceptually:
1
2
3
4
5
6
7
Request: abc123

API 100ms
└── Auth 80ms
└── Book Service 120ms
└── Recommendation 7.4s
└── External API 7.2s
Now the bottleneck becomes obvious.
The API wasn't necessarily slow.
The recommendation service was waiting on an external dependency.

Build a Timeline During an Incident

When debugging a production problem, create a timeline.
For example:
1
2
3
4
5
6
7
8
9
10:02 — Deployment started
10:05 — Deployment completed
10:07 — CPU increased
10:08 — Database connections increased
10:10 — Latency increased
10:11 — 5xx errors increased
10:12 — CloudWatch alarm triggered
10:15 — Rollback started
10:17 — Error rate returned to normal
This is far more useful than saying:
"The server suddenly stopped working."
A timeline helps separate:
What happened
from
What caused it.

Avoid Changing Five Things at Once

When production is broken, there is a temptation to start changing everything.
For example:
1
2
3
4
5
6
7
8
9
Change security group
↓
Restart EC2
↓
Change database settings
↓
Modify application code
↓
Restart ALB
Eventually the application works.
But what fixed it?
You don't know.
A better process is:
1
2
3
4
5
6
7
8
9
10
11
Observe
↓
Form a hypothesis
↓
Test
↓
Verify
↓
Fix
↓
Monitor
For example:
Observation: Database connection count is unusually high.
Hypothesis: The application is not releasing connections correctly.
Test: Inspect connection behavior and application logs.
Verification: Connection count increases continuously with requests.
Fix: Correct connection handling.
Monitor: Confirm connection count returns to normal.
This produces knowledge that can prevent the same incident from happening again.

Design for Debuggability

Observability shouldn't be something you add after the application breaks.
It should be part of the architecture.
When designing an AWS application, ask:

Can I identify a failed request?

Use request IDs or correlation IDs.

Can I identify which service is slow?

Use metrics and distributed tracing.

Can I determine when the problem started?

Use time-series metrics and logs.

Can I identify what changed?

Track deployments and infrastructure changes.

Can I detect the problem automatically?

Create meaningful alarms.

Can I investigate the problem without logging into every server?

Centralize logs and monitoring.
These questions should be considered before production traffic arrives.

What Good Observability Looks Like

A useful production environment might provide a flow like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Incident
|
v
CloudWatch Alarm
|
v
Unified Timeline
|
+---------+---------+
| | |
v v v
Logs Metrics Traces
| | |
+---------+---------+
|
v
Root Cause
|
v
Fix
|
v
Prevention
The important part isn't the individual AWS service.
It's the relationship between the information they provide.

From Monitoring to Observability

There is an important distinction.
Monitoring generally tells you that something is wrong.
For example:
CPU is above 90%.
Observability helps you investigate why.
For example:
CPU increased immediately after deployment v2.4.1, request latency increased at the same time, and the increase is concentrated in the recommendation service.
The second answer is much more useful.
Monitoring tells you:
Something happened.
Observability helps you answer:
Why did it happen?

The Real Goal: Reduce Guesswork

Production incidents are stressful because uncertainty is expensive.
You don't want your debugging process to look like:
1
2
3
4
5
6
7
8
9
10
11
Something broke.
↓
Maybe EC2?
↓
Maybe RDS?
↓
Maybe networking?
↓
Maybe application code?
↓
Let's restart everything.
You want:
1
2
3
4
5
6
7
8
9
10
11
12
13
Something broke.
↓
Which request?
↓
Which component?
↓
What changed?
↓
What evidence supports the hypothesis?
↓
Fix
↓
Verify
That's the difference between guessing and engineering.
# amazon-cloudwatch# monitoring# monitoring-observability
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article