Day 9: Monitoring EKS
As part of my AWS DevOps Community Builders learning journey, Day 9 focused on monitoring Amazon EKS and understanding how observability helps us detect, troubleshoot, and respond to issues in Kubernetes environments.

Today's main focus was:
CloudWatch → Prometheus → Grafana → Metrics → Logs → Alerts
Monitoring is not just about checking whether Pods are running.
It is about understanding what is happening across the entire application and infrastructure.
What I Learned
- Amazon CloudWatch
- CloudWatch Logs
- CloudWatch Metrics
- Amazon Managed Service for Prometheus
- Amazon Managed Grafana
- Kubernetes metrics
- EKS monitoring
- Application monitoring
- Infrastructure monitoring
- Container logs
- CPU and memory utilization
- Pod health
- Node health
- Alerts
- Dashboards
- Observability
- Troubleshooting production workloads
1. Why Monitoring Matters
Imagine an application is running in EKS.
The Pods may show:
Running
But users might still experience:
- Slow response times
- Application errors
- High CPU usage
- High memory usage
- Network problems
- Database connection failures
This is why simply checking:
kubectl get pods
is not enough for a production environment.
We need visibility into:
Metrics
Logs
Traces
Events
Alerts
This is where observability becomes important.
2. What is Amazon CloudWatch?
Amazon CloudWatch provides monitoring and observability capabilities for AWS resources and applications.
It can help collect and visualize:
- Metrics
- Logs
- Events
- Alarms
For EKS environments, CloudWatch can help monitor cluster and workload behavior.
A simplified architecture is:
EKS
↓
CloudWatch
↓
Metrics + Logs
↓
Dashboard
↓
Alerts
3. EKS Monitoring
A production EKS environment can contain:
EKS Cluster
↓
Nodes
↓
Pods
↓
Containers
↓
Applications
Each layer can produce useful monitoring data.
For example:
Node
→ CPU
→ Memory
→ Disk
Pod
→ CPU
→ Memory
→ Restart Count
Application
→ Request Rate
→ Error Rate
→ Response Time
Database
→ Connections
→ CPU
→ Storage
→ Latency
Monitoring these layers gives us a better understanding of application health.
4. CloudWatch Logs
Application containers generate logs.
For example:
Application Started
Database Connected
Request Received
Request Failed
Application Error
These logs are valuable during troubleshooting.
A typical flow is:
Application Container
↓
Container Logs
↓
CloudWatch Logs
↓
Log Groups
↓
Search / Analysis
↓
Trououbleshooting
Instead of logging into individual nodes to inspect logs manually, centralized logging makes investigation easier.
5. Kubernetes Logs
We can also inspect Pod logs directly using kubectl.
Example:
kubectl logs <pod-name>
For a specific container:
kubectl logs <pod-name> -c <container-name>
Follow live logs:
kubectl logs -f <pod-name>
This is useful when troubleshooting application-level problems.
6. Metrics
Metrics provide numerical information about the health and performance of the system.
Examples include:
CPU Usage
Memory Usage
Pod Restart Count
Network Traffic
Request Count
Error Rate
Response Time
For example:
CPU Usage: 75%
Memory Usage: 82%
Request Rate: 1,200 requests/min
Error Rate: 2%
Response Time: 350 ms
These numbers help us understand whether the application is healthy.
7. Amazon Managed Service for Prometheus
Prometheus is widely used for collecting and querying metrics from Kubernetes environments.
Amazon Managed Service for Prometheus provides a managed Prometheus-compatible monitoring service.
A simplified flow is:
EKS
↓
Prometheus Metrics
↓
Amazon Managed Service for Prometheus
↓
Queries
↓
Dashboards / Alerts
This allows teams to collect Kubernetes and application metrics without managing the complete Prometheus infrastructure themselves.
8. Amazon Managed Grafana
Grafana provides dashboards for visualizing metrics.
A typical architecture can look like:
EKS
↓
Prometheus
↓
Amazon Managed Service for Prometheus
↓
Amazon Managed Grafana
↓
Dashboard
A dashboard can display:
- CPU usage
- Memory usage
- Pod count
- Node health
- Request rate
- Error rate
- Application latency
This gives engineers a visual overview of the environment.
9. CloudWatch vs Prometheus vs Grafana
Each tool has a different role.
CloudWatch:
AWS monitoring, metrics, logs, alarms, and observability.
Prometheus:
Metrics collection and querying, especially useful for Kubernetes and application metrics.
Grafana:
Visualization and dashboards.
A simple way to remember:
CloudWatch → Monitor AWS
Prometheus → Collect and query metrics
Grafana → Visualize metrics
10. Monitoring Architecture
A simplified EKS monitoring architecture is:
EKS CLUSTER
|
+----------+----------+
| |
↓ ↓
Metrics Logs
| |
↓ ↓
Prometheus CloudWatch
|
↓
Managed Prometheus
|
↓
Managed Grafana
|
↓
Dashboards
|
↓
AlertsCloudWatch can also provide metrics, logs, alarms, and observability capabilities across AWS resources.
11. Monitoring Pods
Pod monitoring is important because application failures often start at the workload level.
Useful information includes:
- Pod status
- CPU usage
- Memory usage
- Restart count
- Container status
- Readiness
- Liveness
- Application logs
Useful command:
kubectl get pods
More detailed information:
kubectl describe pod <pod-name>
Logs:
kubectl logs <pod-name>
These commands are useful for first-level troubleshooting.
12. Monitoring Nodes
EKS nodes also need monitoring.
Important metrics include:
- CPU utilization
- Memory utilization
- Disk usage
- Network traffic
- Node status
- Pod capacity
Useful command:
kubectl get nodes
Detailed information:
kubectl describe node <node-name>
A node with high resource utilization can affect multiple Pods running on that node.
13. Application Monitoring
Infrastructure health does not always mean application health.
For example:
CPU: Normal
Memory: Normal
Pods: Running
But:
API Response Time: High
Error Rate: High
This means the application can still have a problem even though the infrastructure looks healthy.
Application-level metrics are therefore important.
Useful metrics include:
- Request rate
- Error rate
- Response time
- HTTP status codes
- Database latency
- External API latency
14. Alerts
Monitoring becomes much more useful when it can notify engineers about problems.
For example:
CPU > 80%
Memory > 85%
Pod restart count increases
Error rate > 5%
Application latency increases
Database connections reach a critical level
The basic idea is:
Metric
↓
Threshold
↓
Alert
↓
Engineer
↓
Investigation
↓
Action
Alerts should be meaningful and actionable.
Too many unnecessary alerts can create alert fatigue.
15. Example Production Scenario
Imagine an application running on EKS suddenly becomes slow.
Users report:
"The website is taking too long to load."
Instead of guessing, we can investigate systematically.
Step 1:
Check Pods.
kubectl get pods
Step 2:
Check Pod logs.
kubectl logs <pod-name>
Step 3:
Check resource usage.
kubectl top pods
Step 4:
Check nodes.
kubectl get nodes
Step 5:
Check CloudWatch metrics and logs.
Step 6:
Check application metrics in Prometheus.
Step 7:
Open Grafana dashboards.
Step 8:
Check error rate and response time.
This gives us a structured troubleshooting process.
16. Observability
Monitoring tells us what is happening.
Observability helps us understand why it is happening.
The three major pillars are:
Metrics
Logs
Traces
Metrics tell us:
"What is happening?"
Logs tell us:
"What happened?"
Traces help answer:
"Where did the request spend time?"
Together they provide better visibility into distributed applications.
17. EKS Production Observability
A production architecture can look like:
Users
↓
ALB
↓
EKS
↓
Application Pods
↓
Metrics + Logs
↓
CloudWatch + Prometheus
↓
Grafana
↓
Dashboards
↓
Alerts
This allows DevOps and platform teams to monitor the complete application environment.
18. Monitoring Stack
A possible AWS-based monitoring stack is:
Amazon EKS
↓
CloudWatch
+
Amazon Managed Service for Prometheus
↓
Amazon Managed Grafana
↓
Dashboards
↓
Alerts
Each service contributes to a different part of observability.
19. Important Kubernetes Commands
Check Pods:
kubectl get pods
Check all resources:
kubectl get all
Check Services:
kubectl get svc
Check Nodes:
kubectl get nodes
Check Pod details:
kubectl describe pod <pod-name>
Check logs:
kubectl logs <pod-name>
Follow logs:
kubectl logs -f <pod-name>
Check resource usage:
kubectl top pods
Check node resource usage:
kubectl top nodes
These commands are useful for day-to-day EKS troubleshooting.
20. Monitoring Checklist
For a production EKS environment, I would monitor:
☑ Cluster health
☑ Node health
☑ Pod health
☑ CPU utilization
☑ Memory utilization
☑ Disk utilization
☑ Network traffic
☑ Pod restarts
☑ Application errors
☑ Request rate
☑ Response time
☑ Database performance
☑ Logs
☑ Alerts
☑ Availability
☑ Security events
21. Key Takeaway
The biggest lesson from Day 9 is:
Running an application is only the beginning.
We also need to know:
Is it healthy?
Is it performing well?
Are users experiencing errors?
Are resources being exhausted?
What changed?
Where is the problem?
Why did it happen?
Monitoring and observability help answer these questions.
A simplified flow is:
EKS
↓
Metrics + Logs
↓
CloudWatch + Prometheus
↓
Grafana
↓
Dashboards
↓
Alerts
↓
Troubleshooting
↓
Action
Connecting Day 8 and Day 9
Day 8 focused on production architecture:
Route 53
↓
ALB
↓
Ingress
↓
EKS
↓
Application Pods
↓
RDS
Day 9 adds observability:
EKS
↓
Metrics
↓
Logs
↓
CloudWatch
↓
Prometheus
↓
Grafana
↓
Alerts
Now the architecture is not only designed to run the application, but also to help us understand its health and behavior.
My Day 9 Takeaway
Today I learned that a production-ready EKS environment needs more than deployment and networking.
It also needs visibility.
CloudWatch helps monitor AWS resources and collect logs and metrics.
Prometheus helps collect and query Kubernetes and application metrics.
Grafana helps visualize those metrics through dashboards.
Alerts help teams respond when important thresholds or conditions are reached.
The ultimate goal is:
Monitor → Detect → Investigate → Respond → Improve
What's Next?
Day 10: DevSecOps Pipeline
Topics:
- SonarQube
- Code Quality
- Vulnerability Scanning
- Container Security
- Dependency Scanning
- Security Gates
- Quality Gates
- GitLab CI/CD
- Secure Docker Images
- Secure EKS Deployments
The journey continues:
Learn → Build → Secure → Automate → Deploy → Monitor → Improve
#AWS #AWSCommunityBuilders #DevOps #AmazonEKS #CloudWatch #Prometheus #Grafana #Kubernetes #Observability #Monitoring #DevSecOps #CloudComputing #DevOpsJourney #LearningInPublic
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article