The Amazon Great Indian Festival Started: How AWS EKS Handles Millions of Users
How can AWS EKS handle sudden traffic spikes during a major sale? This post explains how HPA, Karpenter, and EKS Auto Mode scale Pods and compute capacity up during high demand and scale them down when traffic drops.
The Amazon Great Indian Festival has started.
Millions of customers are visiting Amazon.in to search for products, compare prices, check offers, add products to their carts, and place orders.
From a customer's perspective, everything looks simple.
You search for a product.
You open the product page.
You click Add to Cart.
You complete the payment.
But behind those few clicks, a huge number of requests can be moving through different application services.
This made me think about an interesting cloud engineering question:
How can an application handle a sudden increase in traffic without running its maximum infrastructure all the time?
One of the technologies that can be used to build this type of elastic architecture is Amazon EKS (Elastic Kubernetes Service).
Important: Amazon's actual production architecture is proprietary. This article is a conceptual example of how an EKS-based application can handle sudden traffic increases using Kubernetes and AWS autoscaling capabilities.
What happens when traffic suddenly increases?
Imagine an application normally has five Pods running.
1
2
3
4
5
6
7
8
9
10
11
12
13
Users
|
v
Load Balancer
|
+-----------+-----------+
| | |
v v v
Pod 1 Pod 2 Pod 3
|
Pod 4
|
Pod 5
During normal traffic, these Pods may be enough.
Now imagine a large sale starts.
More customers arrive.
More product searches happen.
More cart requests are generated.
More orders are placed.
The traffic might conceptually change like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
Normal traffic
|
v
10K
|
v
50K
|
v
200K
|
v
500K+
The numbers above are only an example to explain the scaling problem.
The important point is that traffic is not constant.
If we permanently provision infrastructure for the peak, we may waste resources during normal periods.
If we provision only for normal traffic, we may not have enough capacity during the spike.
This is where autoscaling becomes important.
EKS doesn't mean "one server"
A common misunderstanding is to think of Kubernetes as one large server.
It is better to think of an EKS cluster as a collection of compute resources where Kubernetes schedules application workloads.
A simplified architecture could look like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Users
|
v
CloudFront
|
v
Application Load Balancer
|
v
EKS Cluster
|
+------------+------------+
| | |
v v v
Product Cart Search
Pods Pods Pods
| | |
+------------+------------+
|
+---------+---------+
| |
v v
Cache Database
The load balancer distributes requests across available application Pods.
Kubernetes manages those Pods.
And autoscaling can adjust the number of Pods and the compute capacity available to run them.
How does EKS scale the application?
This is where Horizontal Pod Autoscaler (HPA) comes in.
HPA can automatically adjust the number of replicas in a Kubernetes workload based on metrics such as CPU and memory, and it can also use custom or external metrics.
For example:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: shopping-api
spec:
minReplicas: 5
maxReplicas: 100
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: shopping-api
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 60
Here we are saying:
1
2
3
Minimum Pods = 5
Maximum Pods = 100
Target CPU = 60%
So during normal traffic:
1
5 Pods
When demand increases:
1
2
3
4
5
6
7
5 Pods
↓
10 Pods
↓
20 Pods
↓
40 Pods
The exact number isn't fixed. HPA calculates the desired replica count based on the configured metrics and target.
What is actually happening?
Think about the process like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Traffic increases
|
v
Pod utilization
increases
|
v
HPA checks
metrics
|
v
More replicas required?
|
Yes
|
v
More Pods created
|
v
Traffic distributed
across more Pods
So instead of manually watching the application and creating Pods, Kubernetes can respond to changing demand.
AWS specifically documents HPA as a way to scale applications out during increased demand and scale them back in when the resources are no longer needed.
But there is another problem
Let's say HPA decides that the application needs 50 additional Pods.
What if the existing EKS nodes are already full?
For example:
1
2
3
Node 1 → CPU almost full
Node 2 → CPU almost full
Node 3 → CPU almost full
Now Kubernetes has more Pods that need to run, but there isn't enough available capacity.
This is where node autoscaling becomes important.
Scaling the Pods is only half of the story
There are actually two different scaling problems:
1. Application scaling
How many Pods do we need?
1
2
3
HPA
↓
More Pods
2. Infrastructure scaling
How much compute capacity do we need?
1
2
3
Karpenter / Cluster Autoscaler / EKS Auto Mode
↓
More Nodes
AWS recommends using a workload autoscaler such as HPA together with node autoscaling approaches such as Karpenter or Cluster Autoscaler.
Karpenter: providing capacity when Pods need it
Karpenter watches for Pods that cannot be scheduled because the cluster doesn't have enough suitable capacity.
A simplified flow is:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Traffic increases
|
v
HPA increases replicas
|
v
New Pods are Pending
|
v
Not enough Node capacity
|
v
Karpenter provisions capacity
|
v
New Node becomes available
|
v
Pending Pods get scheduled
Karpenter is designed to provision and deprovision nodes according to the scheduling requirements of workloads.
This means we don't necessarily need to keep a huge number of EC2 instances running just in case traffic increases.
EKS Auto Mode can automate more of this
Another option available in Amazon EKS is EKS Auto Mode.
EKS Auto Mode extends AWS management into parts of the cluster infrastructure, including compute autoscaling and load balancing.
AWS documents that Auto Mode can add nodes when Pods cannot fit on existing nodes and remove or consolidate nodes when they are no longer needed.
Conceptually:
1
2
3
4
5
6
7
8
9
10
11
12
13
EKS
|
+--------+--------+
| |
v v
Application Infrastructure
Scaling Scaling
| |
v v
HPA EKS Auto Mode
|
v
Node capacity
This can reduce the amount of infrastructure management required by the team.
What happens when traffic reaches the peak?
Let's put everything together.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
USERS
|
v
CloudFront
|
v
Application Load Balancer
|
v
EKS Cluster
|
+----------------+----------------+
| | |
v v v
Product Pods Cart Pods Search Pods
| | |
+----------------+----------------+
|
v
HPA
|
Traffic increases
|
v
More Pods needed
|
v
Existing capacity?
/ \
Yes No
| |
| v
| Node Autoscaling
| |
| v
| More capacity
| |
+--------------+
|
v
More Pods
Now the application has more capacity to handle the increased workload.
But adding Pods isn't enough
This is an important lesson.
Imagine we have:
1
2
3
4
100 Pods
|
v
One database
If every Pod starts sending huge numbers of queries to the same database, the database can become the bottleneck.
So large-scale applications need to scale multiple layers.
For example:
1
2
3
4
5
6
7
8
9
10
11
12
13
Users
|
v
Load Balancer
|
v
Application Pods
|
+------> Cache
|
+------> Database
|
+------> Queue
Caching can reduce repeated database reads.
Queues can help decouple workloads.
Databases may need their own scaling strategy.
And observability is needed to understand what is actually becoming a bottleneck.
EKS solves container orchestration and scaling problems, but it doesn't automatically make every downstream dependency infinitely scalable.
What happens after the traffic goes down?
This is my favorite part of autoscaling.
Autoscaling isn't only about adding capacity.
It is also about giving capacity back when you don't need it.
Suppose traffic starts falling after the peak:
1
2
3
4
5
6
7
8
9
500K requests
↓
300K
↓
100K
↓
30K
↓
10K
HPA can reduce the number of application replicas according to its configuration.
For example:
1
2
3
4
5
6
7
100 Pods
↓
70 Pods
↓
40 Pods
↓
15 Pods
Now some nodes may have significantly more unused capacity.
A node autoscaler can then consolidate or remove nodes when the workload no longer requires them.
Karpenter is designed to handle node deprovisioning as workloads change, while EKS Auto Mode can dynamically remove nodes that are no longer needed.
So the flow becomes:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Traffic decreases
|
v
Fewer Pods required
|
v
HPA scales in
|
v
Unused Node capacity
|
v
Node autoscaler
|
v
Nodes removed/consolidated
This is the scale-out → scale-in cycle.
Scaling isn't instantaneous
There is one important thing to remember.
Autoscaling isn't magic.
If traffic suddenly jumps:
1
10K → 500K
the infrastructure doesn't instantly become 50 times larger.
There is a reaction time.
Metrics need to be collected.
HPA needs to evaluate the metrics.
Pods need to start.
Container images may need to be pulled.
If new nodes are required, they need to be provisioned and become ready.
That's why capacity planning is still important for predictable events.
For a known traffic spike, teams can start with a higher baseline and let autoscaling handle additional demand.
A simple mental model
When I think about EKS autoscaling, I use this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
TRAFFIC
|
v
Load Balancer
|
v
EKS Pods
|
v
HPA
|
More Pods needed?
/ \
No Yes
| |
| v
| Node capacity?
| / \
| Yes No
| | |
| | v
| | Karpenter /
| | EKS Auto Mode
| | |
| +---------+
| |
+-----------------+
|
v
Application
And when traffic goes down:
1
2
3
4
5
6
7
8
9
10
11
12
13
Traffic decreases
|
v
HPA scales Pods down
|
v
Unused capacity
|
v
Node autoscaling
|
v
Nodes can be removed
The real lesson
When we see millions of users shopping during a major sale, it is easy to think:
"They must be running a huge number of servers."
The more interesting engineering question is:
How do you make the infrastructure adapt to demand?
You don't want to permanently run your peak capacity if you don't need it.
You also don't want insufficient capacity when users suddenly arrive.
The goal is elasticity.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Traffic
|
+--------+--------+
| |
v v
Traffic Up Traffic Down
| |
v v
More Pods Fewer Pods
| |
v v
More Compute Less Compute
| |
+--------+--------+
|
v
Elastic System
And that's what makes EKS interesting for high-traffic applications.
HPA can scale the application.
Karpenter or other node autoscaling mechanisms can provide the compute capacity.
EKS Auto Mode can automate additional infrastructure operations.
Load balancing distributes traffic.
Caching and databases handle the data layer.
Together, these components allow an application to respond to changing demand rather than being locked into a fixed amount of infrastructure.
The next time you click "Buy Now" during a massive sale, remember that behind that single click there may be an entire distributed system deciding where your request should go and how much capacity the platform needs at that moment.
That's the interesting side of cloud engineering: the infrastructure has to keep changing while the user experience should feel like nothing changed at all.
Key Takeaways
- EKS provides managed Kubernetes for running containerized workloads.
- HPA can increase or decrease application Pod replicas based on configured metrics.
- Karpenter can provision and deprovision compute capacity based on workload requirements.
- EKS Auto Mode can automate compute autoscaling and other infrastructure capabilities.
- Scale-out handles increased demand.
- Scale-in reduces unused capacity after demand falls.
- Autoscaling should be combined with capacity planning, load balancing, caching, database scaling, and observability for high-traffic applications.
Scale when users arrive.
Scale down when they leave.
Keep the application available without permanently running peak capacity.
Scale down when they leave.
Keep the application available without permanently running peak capacity.
That's the real idea behind elastic infrastructure.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article