AWS Builder Center

Builder Center

AWS DevOps Article: What Really Happens When a Kubernetes Deployment Fails on AWS EKS?

One of the most important DevOps skills is not just deploying applications, but knowing how to troubleshoot them when something goes wrong. When an application runs on Amazon EKS, a failure can come from many different layers — Kubernetes, containers, networking, storage, IAM, AWS infrastructure, or the application itself.

🏗️ Understanding the Troubleshooting Flow

A typical EKS application can look like:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Developer
↓
GitHub
↓
GitHub Actions
↓
Docker Build
↓
Amazon ECR
↓
Amazon EKS
↓
Kubernetes
↓
Ingress / Load Balancer
↓
Application
↓
Database / Other Services
When something breaks, randomly checking everything is not the best approach.
Instead, troubleshoot layer by layer.

🔍 1. Check the Pod Status

Start with the Kubernetes workload.
1
kubectl get pods -n <namespace>
Common states include:
  • Running
  • Pending
  • CrashLoopBackOff
  • ImagePullBackOff
  • ErrImagePull
Each status gives you a starting point for investigation.

🐳 2. Check Container Logs

If the pod is running but the application is failing:
1
kubectl logs <pod-name> -n <namespace>
For example, you might discover:
1
Database connection refused
Now the problem may not be Kubernetes itself — it could be the application's database configuration or connectivity.

📋 3. Check Kubernetes Events

Events are extremely useful for understanding scheduling and deployment problems.
1
kubectl describe pod <pod-name> -n <namespace>
You may find issues related to:
  • Insufficient CPU or memory
  • Image pulling
  • Mount failures
  • Scheduling
  • Health checks
  • Permissions

🌐 4. Check Networking

If the application is healthy internally but users cannot access it, investigate:
Ingress → Load Balancer → Service → Pod
Useful commands include:
1
2
3
kubectl get ingress
kubectl get svc
kubectl get endpoints
At this stage, AWS networking can also become important.
You may need to investigate:
  • Load Balancer configuration
  • Security Groups
  • Subnets
  • Route tables
  • Target health

💾 5. Check Storage

Applications that use persistent storage can fail because of storage-related issues.
Check:
1
2
kubectl get pvc
kubectl get pv
A Pending PersistentVolumeClaim can indicate that Kubernetes could not provision or attach the required storage.

🔐 6. Check IAM and Permissions

AWS services and Kubernetes workloads often interact with AWS resources.
If an application cannot access an AWS service, investigate the relevant IAM configuration.
For example:
1
2
3
4
5
Pod
↓
IAM Role
↓
AWS Service
Incorrect permissions can cause failures even when the Kubernetes workload itself appears healthy.

📊 7. Use Observability

Troubleshooting becomes much easier when metrics and logs are centralized.
A typical monitoring stack can look like:
1
2
3
4
5
Kubernetes
↓
Prometheus
↓
Grafana
And for logs:
1
2
3
4
5
6
7
Kubernetes
↓
Filebeat
↓
Elasticsearch
↓
Kibana
This allows engineers to investigate both what is happening and why it happened.

🧠 The DevOps Mindset

The biggest lesson is:
Don't immediately restart everything. Find the failure layer first.
A simple troubleshooting approach is:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Pod Status
↓
Pod Logs
↓
Kubernetes Events
↓
Service / Ingress
↓
Networking
↓
Storage
↓
IAM
↓
AWS Infrastructure
↓
Application Dependencies
This approach helps reduce guesswork and makes troubleshooting more systematic.

🚀 What I Learned

Working with EKS has taught me that Kubernetes knowledge is not only about knowing commands.
It's about understanding how different layers interact:
AWS Infrastructure + Kubernetes + Containers + Networking + IAM + Observability
A successful deployment is only one part of DevOps.
Being able to understand, troubleshoot, and recover from failures is equally important.

💬 Let's Discuss

If your EKS application suddenly becomes unavailable, what would you check first?
1️⃣ Pod status
2️⃣ Application logs
3️⃣ Kubernetes events
4️⃣ Load Balancer / Ingress
5️⃣ AWS networking
Share your approach in the comments. 👇
If you found this useful, Like 👍, Comment 💬, and Share 🔄 so other AWS and DevOps learners can benefit too!
# amazon-eks
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article