AWS Builder Center

Scaling GPU nodes to zero on EKS: what it cost me, and what it saved

A student project that analyzes pre/post-disaster satellite imagery on EKS, where GPU nodes sit at zero when idle. How a CPU screening stage cuts GPU invocations, how Terraform and ArgoCD wire it together, and the cold-start trade-off that came with it.

The problem: GPUs bill you for doing nothing

I built CORE (Cloud-Optimized Resource-Efficient Vision System), a pipeline that compares satellite imagery from before and after a disaster, detects damaged areas, and classifies the disaster into one of seven types — wildfire, flood, hurricane, and so on.
The model work was the fun part. The infrastructure was where I actually got stuck.
Disaster imagery analysis is bursty by nature. Nothing happens for days, then a large batch of imagery arrives at once. But a GPU instance doesn't care whether it's inferring or idling — it bills the same either way. For a student running this out of pocket, an always-on GPU node was not a real option.
So the design constraint became: the cluster must hold zero GPU nodes when no work is in flight.
That constraint ended up shaping the entire architecture, not just the node group config.

Stage 1: don't call the GPU unless you have to

The obvious move is autoscaling. The less obvious move — and the one that mattered more — was reducing how often the GPU needs to run at all.
Most tiles in a satellite image are unremarkable. Forest that is still forest. Ocean that is still ocean. Sending all of them to a ResNet-101 is paying premium compute to confirm nothing happened.
So the pipeline runs in three stages:
  1. CPU screening. A lightweight model scores each image tile for likelihood of damage. This runs on ordinary CPU nodes that are always up and cheap. The risk scoring borrows from UNDRR's national disaster risk assessment indicators rather than a threshold I invented.
  1. Priority queue. Tiles are ordered by that risk score, so the highest-signal tiles reach the GPU first.
  1. GPU inference. ResNet-101 handles detection, then classification into the seven disaster types.
The screening stage hits Macro F1 0.741 — good enough to filter confidently, not good enough to be the final answer. The detection and classification stage lands at Macro F1 0.669, and GPU inference itself runs at roughly 22ms.
That 0.741 is doing the real cost work. Every tile it filters out is a GPU-second I don't pay for.

Stage 2: the node group that starts at zero

Everything is defined in Terraform. The relevant part is that the GPU node group's minimum size is zero, not one:
hcl
resource "aws_eks_node_group" "gpu" {
cluster_name = aws_eks_cluster.core.name
node_group_name = "gpu-inference"
instance_types = ["g4dn.xlarge"]

scaling_config {
desired_size = 0
min_size = 0
max_size = 3
}

labels = {
"workload" = "gpu-inference"
}

taint {
key = "nvidia.com/gpu"
value = "true"
effect = "NO_SCHEDULE"
}
}
The taint matters as much as the scaling config. Without it, ordinary pods land on the GPU node the moment it appears and keep it alive long after inference finishes. Inference pods carry the matching toleration; nothing else does.
Deployment is GitOps — GitLab CI builds the images, ArgoCD syncs the manifests to the cluster. This turned out to be more valuable than I expected for a solo project. When the GPU node group churns, I want the cluster's desired state to live in Git, not in whatever I typed into the console at 2am.

The part nobody puts in the diagram: cold start

Here is the honest trade-off. A node group at zero means the first request after an idle period pays for:
  • node provisioning
  • kubelet registration and the NVIDIA device plugin coming up
  • pulling the inference image, which is not small when CUDA is in it
That is not milliseconds. That is tens of seconds if you do nothing about it.
Two things helped:
Pre-pulling the image. The container image is cached into the AMI so the node isn't downloading multiple gigabytes on its first breath. This was the single biggest win.
Scaling on the queue, not on the pod. The CPU screening stage knows work is coming before the GPU pod is ever scheduled. Triggering the scale-up off queue depth means provisioning starts while screening is still running, so the two overlap instead of stacking.
With both in place, upload to dashboard display sits at roughly 2–3 seconds on a warm path. The genuinely cold path is still slower, and I've made peace with that — for a workload that is idle most of the time, occasional cold-start latency is a fair price for not paying for an idle GPU.

What I'd tell past me

Scale-to-zero is an architecture decision, not a config flag. Setting  min_size = 0  took one line. Making the system tolerate that zero took the screening stage, the queue, the taints, and the image caching.
Cheap compute upstream buys you expensive compute downstream. The CPU model isn't as accurate as the GPU model and doesn't need to be. It just needs to be right about what's boring.
Publish the unflattering numbers. Macro F1 0.669 on detection and classification is not a headline. But a post that only shows the numbers that look good isn't much use to the next person sizing up the same trade-off.


Built as a student project. If you're running a bursty GPU workload on EKS and handling cold start differently, I'd like to hear how — leave a comment.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article