Kubernetes on AWS — the full set
Five articles from the aws-demo repository in one document, generated on 2026-10-02 from
docs/. Sources: the Kubernetes notes and cheat sheets, the minikube and Gateway API docs, Zhao Huabing's controller post, the k8s.af failure stories, the AI-infrastructure draft, the Spring Boot multi-architecture blog, the ticketing and multi-agent pages, and the Replit incident write-up. Each part keeps its own Verified, and not section.Contents
- Kubernetes on AWS, services only
- Kubernetes troubleshooting on EKS
- Kubernetes failure stories on AWS
- From demo to production on AWS
- Multi-agent operations on AWS
Kubernetes on AWS, services only
A Kubernetes course teaches you to build a cluster from parts — kubeadm on EC2, Calico, an NGINX ingress, hostPath volumes, Secrets in base64. On EKS almost every one of those parts is an AWS service or a managed add-on, and the EKS API now exposes control-plane settings the courses describe as fixed: event TTL, the scheduler's scoring strategy, the HPA sync period, a provisioned control-plane tier, and deletion protection. Chapter by chapter, from the notes and cheat sheets this repository's troubleshooting page was built from: what AWS runs, what is still yours, and where this repository already sits.
The study material behind Kubernetes troubleshooting on EKS — a 293-page End-to-End Kubernetes Notes, a 36-page basics notebook, two cheat sheets, the minikube and Gateway API documentation, and a post on how controllers are built — teaches Kubernetes the way most courses do: as parts you assemble. A control plane installed with
kubeadm on EC2, Calico for pod networking, an NGINX ingress, hostPath and awsElasticBlockStore volumes, Secrets in base64, Helm for everything else.This page walks the same chapters and asks, for each part: on AWS, with managed services only, who runs it now? The answer is usually an EKS add-on, a controller AWS publishes, or a field on the EKS API itself — and the field list has grown. The EKS API model (botocore 1.43.107) now accepts, at cluster creation, settings every course describes as out of reach on a managed control plane.
The map
| course chapter | what the notes build | on AWS, services only | still yours |
|---|---|---|---|
| architecture, control plane | kubeadm init on EC2, stacked etcd | Amazon EKS — the API servers, etcd and controllers, with a provisioned scaling tier if you need it | the load you put on the API server |
| cluster setup | minikube, kubeadm, three EC2 instances | EKS via Terraform, or EKS Auto Mode | local development clusters — kind or minikube, no AWS equivalent |
| nodes | EC2 instances you join by hand | managed node groups, Karpenter, Auto Mode node pools, or Fargate | instance choice, AMI family (AL2023 or Bottlerocket) |
| container images | Docker Hub | Amazon ECR, with scan on push | tags and lifecycle |
| pod networking | Calico manifest | Amazon VPC CNI add-on — pods get VPC addresses | subnet sizing, the SNAT setting |
| NetworkPolicy | Calico | the VPC CNI's network policy agent | the policies |
| Services, kube-proxy | iptables rules | the kube-proxy add-on | — |
| cluster DNS | CoreDNS manifest | the CoreDNS add-on | replicas, ndots |
| type: LoadBalancer | a pending external IP | AWS Load Balancer Controller → NLB | scheme, target type |
| Ingress | NGINX ingress controller | AWS Load Balancer Controller → ALB | listener and TLS settings |
| Gateway API | an implementation you pick | AWS Load Balancer Controller (ALB, NLB) north–south; AWS Gateway API Controller for VPC Lattice east–west | routes, ReferenceGrants |
| volumes, PV/PVC | hostPath, in-tree awsElasticBlockStore | EBS CSI (block, zonal), EFS CSI (shared, multi-AZ), FSx for Lustre CSI, Mountpoint for S3 CSI — or Auto Mode block storage | the StorageClass and binding mode |
| ConfigMaps, Secrets | base64 in etcd | Secrets encrypted with KMS; secret values in Secrets Manager, mounted with the Secrets Store CSI provider | which secrets live where |
| RBAC, users | kubeconfig certificates | IAM plus EKS access entries and access policies | the policies |
| pod identity | node instance profile | EKS Pod Identity or IRSA | per-ServiceAccount roles |
| HPA, metrics | metrics-server manifest | metrics-server as an add-on; custom metrics from CloudWatch or Amazon Managed Service for Prometheus | the targets |
| logging, monitoring | Prometheus and Grafana on EC2 | CloudWatch Container Insights add-on, AMP managed collector, Amazon Managed Grafana, control-plane logs | dashboards and alerts |
| StatefulSets for databases | MySQL or PostgreSQL in the cluster | Amazon RDS or Aurora | the decision to keep state out of the cluster |
| Helm, GitOps | helm install from a laptop | EKS add-ons for cluster components; the Argo CD EKS capability for applications | charts and manifests |
| operators, CRDs | a hand-built controller | the ACK and kro EKS capabilities for AWS resources and compositions | your own controllers |
The control plane is no longer a black box
Every course says the managed control plane's flags are not yours to set. The EKS
CreateCluster API now takes:| field | what it sets | why it matters |
|---|---|---|
| kubeApiServerConfig.eventTtl | how long the API server keeps Events | upstream's default is one hour, which is why yesterday's failure has no events by morning |
| kubeApiServerConfig.serviceNodePortRange | the NodePort range | ports that collide with something on the node |
| kubeSchedulerConfig.nodeResourcesFit.scoringStrategy | how the scheduler scores nodes | bin-packing (MostAllocated) instead of spreading, so the autoscaler can empty nodes |
| kubeControllerManagerConfig.horizontalPodAutoscalerControllerConfig | the HPA sync period | how quickly the HPA reacts |
| kubeControllerManagerConfig.podGcControllerConfig | the terminated-pod GC threshold | finished Job pods that clutter the API |
| controlPlaneScalingConfig.tier | standard, tier-xl up to tier-8xl | a provisioned control plane for API load that bursts faster than autoscaling |
| deletionProtection | refuses DeleteCluster until turned off | the Spotify story, prevented by one field |
| upgradePolicy.supportType | STANDARD or EXTENDED | new clusters default to extended support, which is billed; an old version keeps running and keeps costing |
| zonalShiftConfig | Application Recovery Controller zonal shift | moving traffic out of an impaired zone |
Field names and values are from the API model; which Kubernetes versions and regions accept each one is in the EKS user guide.
Three ways to run nodes, and what each takes away
| managed node groups | Karpenter | EKS Auto Mode | |
|---|---|---|---|
| who picks instances | you, per group | Karpenter, per pod | AWS, through built-in node pools |
| who runs the autoscaler | you (Cluster Autoscaler) | you (the controller) | AWS |
| who patches nodes | you, by rolling the group | drift, when the AMI changes | AWS, with a maximum node lifetime |
| load balancers and EBS | controllers you install | controllers you install | built in |
| this repository | today — three t3.medium nodes plus Graviton | Karpenter on AWS (docs/karpenter-on-aws.md) | — |
Auto Mode is the furthest a cluster gets from the course's
kubeadm init: compute, block storage and load balancing are configured on the cluster itself (computeConfig, storageConfig, and kubernetesNetworkConfig.elasticLoadBalancing in the API). The price is control — the node operating system and the controllers are AWS's choice.Networking: the VPC is the cluster network
The notes install Calico and explain overlay routing. On EKS the VPC CNI gives every pod an address from the VPC subnet, so pods are reachable from anything the VPC routes to, security groups can apply to pods, and there is no overlay to debug. The costs move elsewhere: pod density is bounded by the instance's ENIs (17 pods on a
t3.medium without prefix delegation), subnets must be sized for pods rather than nodes, and the CNI translates pod traffic leaving the VPC unless told not to. Kubernetes failure stories on AWS has the EKS migration that broke on exactly that setting.For traffic in, the notes' NGINX ingress becomes the AWS Load Balancer Controller: an
Ingress becomes an ALB, a Service of type: LoadBalancer an NLB, and in IP mode the load balancer targets pods directly. The Gateway API chapter maps to two AWS implementations, and they are different products: the load balancer controller for ALBs and NLBs, and the AWS Gateway API Controller for VPC Lattice, service-to-service across VPCs and accounts. Service mesh on AWS (docs/service-mesh-on-aws.md) chooses the second for east–west.Storage: zonal by default
The notes' PersistentVolume examples use
hostPath and the in-tree awsElasticBlockStore plugin, which was removed in Kubernetes 1.27. On AWS:| need | service | the catch |
|---|---|---|
| a disk for one pod | EBS via the EBS CSI driver, gp3 | zonal: use volumeBindingMode: WaitForFirstConsumer so the volume is created where the pod lands |
| a filesystem many pods share | EFS via the EFS CSI driver | regional and ReadWriteMany; per-GB pricing is higher |
| high-throughput training data | FSx for Lustre | provisioned capacity |
| read-mostly objects | Mountpoint for S3 CSI driver | not POSIX; no in-place writes |
| a database | RDS or Aurora | not a volume at all — which is the point |
Identity: two systems, both IAM
The notes manage users with kubeconfig certificates and RBAC. On EKS, people and pipelines authenticate with IAM, and access entries map IAM principals to Kubernetes permissions — either Kubernetes groups bound by your RBAC, or EKS access policies scoped to the cluster or to namespaces. Set
accessConfig.authenticationMode to API and the aws-auth ConfigMap, whose one typo locks everyone out, is no longer consulted.Pods are the second system: EKS Pod Identity (or IRSA) gives each ServiceAccount its own IAM role, so no pod inherits the node's. This repository uses IRSA for the application and installs the Pod Identity agent add-on (
terraform/main.tf:52).Controllers: the post's machinery, and the managed ones
The controller post builds one from list-watch, an informer, a work queue and a reconcile loop. Every managed piece above is that same machine, run by someone else: the VPC CNI's IP address manager, the load balancer controller, Karpenter. The EKS capabilities take it one step further — ACK turns AWS resources (an S3 bucket, an RDS instance, an SQS queue) into Kubernetes custom resources reconciled by AWS-run controllers; kro composes several resources into one; Argo CD reconciles applications from git. What stays yours is the controller for your own domain, and the post's rules for it: write failures into
status, set observedGeneration, and reconcile from state.What has no AWS service, and should not
Some chapters are Kubernetes itself and stay that way: Pods, Deployments, ReplicaSets, DaemonSets, Jobs, probes, labels and selectors, affinity and topology spread, resource requests, PodDisruptionBudgets, quotas and LimitRanges. No managed service writes them for you. They are the chapters the cheat sheets cover, and the ones the troubleshooting page corrects — CPU limits, readiness versus liveness, why a pod is Pending.
Local clusters also stay outside AWS. This repository's
k8s/local overlay runs the same manifests on kind: no load balancer controller, no IRSA, a NodePort instead of an ALB. A bug that reproduces there is a Kubernetes bug; one that does not is usually one of the AWS services on this page.Where this repository sits
| layer | today | the managed next step |
|---|---|---|
| control plane | EKS, kubernetes_version = "1.30" | a version in standard support; deletionProtection = true; a longer eventTtl |
| add-ons | coredns, kube-proxy, vpc-cni, eks-pod-identity-agent (terraform/main.tf:48–53) | the EBS CSI driver and the CloudWatch observability add-on |
| nodes | managed node groups | Karpenter or Auto Mode |
| ingress | AWS Load Balancer Controller, ALB in IP mode | HTTPS with ACM; a pod readiness gate |
| identity | IRSA, creator admin through an access entry | authenticationMode: API, a named break-glass role |
| delivery | make deploy | the Argo CD capability |
Verified, and not
From the EKS API model (botocore 1.43.107): every
CreateCluster field and enum in the control-plane table, Auto Mode's computeConfig, storageConfig and elasticLoadBalancing, the access-entry and Pod Identity operations, the authentication modes, and the capability types ACK, KRO and ARGOCD.From this repository: the add-ons, node groups, version default, IRSA and the ALB's IP target mode.
From AWS documentation, not checked against the API: which add-ons are published by AWS versus the community, the VPC CNI network policy agent and security groups for pods, the storage drivers' behaviour, Auto Mode's node lifetime and patching, and pricing for extended support and provisioned control-plane tiers. Check the current EKS user guide before relying on any one of them.
From the source notes: the chapter list, as summarized in the troubleshooting page that read them in full. No cluster was created for this page.
Kubernetes troubleshooting on EKS
A field guide that replaces the cheat sheet. One model — the chain of controllers between kubectl apply and a request reaching a pod — and, for each link, where it writes down that it is stuck. A symptom table keyed to the owning controller, the exit codes that name a cause, the four failures this repository has already hit or is set up to hit (a non-numeric USER, a 128Mi limit, 17 pods per t3.medium, and an arm64 node pulling an amd64 image), Gateway API status as the debugger it was designed to be, what kind and minikube can and cannot reproduce, and 31 corrections to the notes and cheat sheets this page was built from.
The runbook (
docs/aws-runbook.md) covers this repository's two AWS-shaped failures — the ALB that never appears and the ALB that returns 503. This page is everything underneath: why a pod is Pending, why it restarts, why a Service has nowhere to send traffic, and how to find out in four commands instead of forty.It was built from a stack of study material — two kubectl cheat sheets, a 36-page Kubernetes Basic to Advance notebook, Aman Pathak's 293-page End-to-End Kubernetes Notes, the minikube and Gateway API documentation, and Zhao Huabing's post on how a Kubernetes controller is built. The study material is good at what objects exist and weak at what to do when one is stuck; between them they have no symptom-to-cause table at all, and about thirty commands or claims that are wrong today. The last section lists them.
The one idea: find the controller that stopped
Kubernetes has no deploy step.
kubectl apply writes an object to the API server and returns. Everything after that is a chain of independent controllers, each watching for objects in a state it is responsible for and moving them one step on. When something does not work, one link in that chain could not do its step — and it almost always wrote down why, on the object it was working on.| step | who does it | what it changes | where it writes a failure | first look |
|---|---|---|---|---|
| 1. accept the object | API server + admission (quota, LimitRange, PodSecurity, webhooks) | stores the Deployment | the kubectl apply error itself | the command's output |
| 2. roll out | Deployment controller | creates / scales a ReplicaSet | Deployment status.conditions (Progressing, Available) | kubectl rollout status |
| 3. make pods | ReplicaSet controller | creates Pod objects | events on the ReplicaSet — e.g. exceeded quota | kubectl describe rs |
| 4. place pods | scheduler | sets spec.nodeName | Pod condition PodScheduled=False, event FailedScheduling | kubectl describe pod |
| 5. give it a network | VPC CNI (aws-node) on the node | assigns a VPC IP | event FailedCreatePodSandBox | kubectl describe pod, aws-node logs |
| 6. run containers | kubelet + containerd | pulls, starts, probes, restarts | container state.waiting.reason, lastState.terminated | kubectl get pod, logs --previous |
| 7. mark ready | kubelet (readiness probe) | Ready=True | Pod condition Ready=False, event Unhealthy | kubectl describe pod |
| 8. be reachable | EndpointSlice controller | adds the pod IP to the Service's slices | an empty slice — no error at all | kubectl get endpointslices |
| 9. route to it | kube-proxy (ClusterIP) or AWS Load Balancer Controller (ALB, IP mode) | iptables rules / target group registration | controller logs; target health in AWS | runbook (docs/aws-runbook.md) |
Two things in that table are worth fixing in your head, because both notebooks get them wrong. The scheduler does not create pods — the ReplicaSet controller does; the scheduler only picks a node for pods that already exist. kube-proxy does not assign pod IPs — the CNI does; on EKS that is the VPC CNI, and the IPs are real VPC addresses.
The model also tells you what to ignore. A pod that is
Pending has not reached step 6, so its logs are empty and kubectl logs is the wrong command. A Service with no endpoints is a step-8 problem, so the ALB's 503 is a symptom, not the fault.Four commands that find it
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# 1. Where in the chain is it? Status, restarts, node, IP in one line.
kubectl -n whereami get deploy,rs,pod -o wide
# 2. What did the stuck controller say? Events for one object, newest last.
kubectl -n whereami events --for pod/whereami-7d9c8b6f5-x2k4q
kubectl -n whereami describe pod whereami-7d9c8b6f5-x2k4q # same events, plus conditions
# 3. Has the controller even seen my last change?
kubectl -n whereami get deploy whereami \
-o jsonpath='{.metadata.generation} {.status.observedGeneration}{"\n"}'
# 4. Why did the previous container die? (The current one has not died yet.)
kubectl -n whereami logs whereami-7d9c8b6f5-x2k4q --previous
kubectl -n whereami get pod whereami-7d9c8b6f5-x2k4q \
-o jsonpath='{.status.containerStatuses[*].lastState.terminated}{"\n"}'
Command 3 is the one nobody uses and the one that settles arguments. If
observedGeneration is behind generation, the controller has not processed your edit — the problem is the controller (crashed, leader election stuck, rate-limited), not your spec. Every well-behaved controller, including the Gateway API implementations below, reports it.Events expire — the API server keeps them for one hour by default — so on a failure from last night,
describe shows nothing and the evidence is in whatever you ship events to. On EKS, the control-plane logs in CloudWatch (if enabled) cover the API server and scheduler, not events; an events exporter is the gap. EKS now lets you lengthen the window itself: the cluster's kubeApiServerConfig.eventTtl sets the API server's event TTL.Symptom → controller → cause
| what you see | stuck at | first command | usual causes | the EKS twist |
|---|---|---|---|---|
| Deployment exists, no pods at all | 1 or 3 | describe rs | ResourceQuota rejects the pod (must specify limits.cpu), PodSecurity rejects it, a webhook times out | none — but this is the one case where get pods shows nothing and people look in the wrong place |
| Pending, FailedScheduling: Insufficient cpu | 4 | describe pod | requests exceed free allocatable on every node | Cluster Autoscaler or Karpenter should add a node; if neither runs, nothing will |
| Pending, Too many pods | 4 | describe node → Allocatable: pods | the node's pod limit | ENI-based: 17 on a t3.medium, whatever the CPU says — see below |
| Pending, didn't match Pod's node affinity/selector | 4 | describe pod | a nodeSelector / affinity no node satisfies, or a taint with no toleration | an arch selector with no Graviton group, or the reverse |
| Pending, volume node affinity conflict | 4 | describe pvc, describe pv | an EBS volume in a different AZ from every node that could take the pod | EBS is zonal; use volumeBindingMode: WaitForFirstConsumer |
| ContainerCreating for minutes | 5 | describe pod → FailedCreatePodSandBox | CNI cannot get an IP | subnet out of IPs, or aws-node unhealthy: kubectl -n kube-system logs -l k8s-app=aws-node |
| ErrImagePull / ImagePullBackOff | 6 | describe pod | wrong tag, no pull permission, registry unreachable | node role lacks ECR read; private subnets with no NAT and no ECR/S3 endpoints; image in another region |
| CreateContainerConfigError | 6 | describe pod | a referenced ConfigMap / Secret / key missing; runAsNonRoot with a non-numeric image user | this repository has hit the second one — see below |
| CrashLoopBackOff | 6 | logs --previous, lastState.terminated | the process exits; read the exit code | exec format error = wrong CPU architecture |
| OOMKilled, exit 137 | 6 | lastState.terminated.reason | working set over the memory limit | JVMs and Go heaps sized off node RAM, not the limit |
| Running, 0/1 READY | 7 | describe pod → Unhealthy | readiness probe failing; the app is up but says it is not ready | the ALB health check uses the same /readyz, so this is the 503 |
| Running, 1/1, Service unreachable | 8 | get endpointslices -l kubernetes.io/service-name=<svc> | Service selector does not match pod labels; targetPort names a port the container does not declare | none |
| Deployment never finishes | 2 | get deploy -o jsonpath='{.status.conditions}' | Progressing=False, reason=ProgressDeadlineExceeded | nothing rolls back; you rollout undo or fix forward |
| Node NotReady | node | describe node → conditions | kubelet down, disk / memory / PID pressure, CNI not ready | aws-node crash-looping takes every new pod on that node with it |
| stuck Terminating | GC | get <obj> -o jsonpath='{.metadata.finalizers}' | a finalizer whose controller is gone | removing the finalizer by hand skips the cleanup it guards — an ALB, an EBS volume |
Exit codes are a diagnosis
| exit code | means | look at |
|---|---|---|
| 0 | the process finished — for a long-running server, that is the bug | the command; a Deployment pod should never exit 0 |
| 1 | the application failed | logs --previous |
| 126 | the entrypoint is not executable | file mode in the image |
| 127 | the entrypoint was not found | command: typo; a distroless image has no sh or bash |
| 137 | SIGKILL | reason: OOMKilled if memory; otherwise a liveness failure that outlived the grace period, or eviction |
| 139 | SIGSEGV | native code, or the wrong libc |
| 143 | SIGTERM | normal on shutdown; a crash loop of 143s means liveness is killing a slow starter — add a startupProbe |
Four failures this repository is set up for
1. CreateContainerConfigError from a non-numeric user — already hit
k8s/base/deployment.yaml records it in a comment: the distroless image declares USER nonroot by name, the pod sets runAsNonRoot: true, and the kubelet refuses to start a container whose user it cannot prove is not 0:1
2
Error: container has runAsNonRoot and image has non-numeric user (nonroot),
cannot verify user is non-root
The fix in the manifest is
runAsUser: 65532. It is the textbook step-6 failure: the pod is scheduled and has an IP, so get pod looks almost fine, and the reason is only in the container's waiting state and its events.Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article