
How to Prevent Cluster Autoscaler from Blocking Node Scale-Down Due to emptyDir Volumes on Amazon EKS
Learn why Cluster Autoscaler skips Amazon EKS nodes using emptyDir volumes and how an application-level safe-to-evict-local-volumes annotation enables safe node scale-down without modifying the Cluster Autoscaler configuration.
Introduction
While working on an Amazon EKS production cluster, we faced an interesting issue.
Our Java application was configured to generate heap dumps whenever an OutOfMemoryError occurred. To store these heap dumps, we used a Kubernetes emptyDir volume because the files were only needed temporarily for troubleshooting.
The application worked exactly as expected. Heap dumps were generated successfully whenever required.
However, after enabling this configuration, we noticed that some EC2 worker nodes were no longer being removed even when they were underutilized.
CPU utilization was low.
Memory utilization was low.
Pods had already moved to other nodes.
But the worker node continued running.
After investigating the logs, we found that Cluster Autoscaler was skipping nodes because one of the application pods was using local storage through an emptyDir volume.
This article explains why this happens, how it affects Amazon EKS, and why adding an application-level annotation is a safer solution than changing the Cluster Autoscaler configuration.

Understanding the Architecture
The issue involved several AWS and Kubernetes components working together.
- Amazon EKS
- Amazon EC2 Managed Node Groups
- Kubernetes Cluster Autoscaler
- Java Application
- Kubernetes Deployment
- emptyDir Volume
- Helm
Although the issue appears to be related to autoscaling, the root cause is actually Kubernetes handling of local storage.
Why We Used emptyDir
Our Java application was configured with the following JVM options:
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/javaheapdump
A simple emptyDir volume was mounted into the container.
volumes:
- name: javaheapdump-volume emptyDir: {}
volumeMounts:
- name: javaheapdump-volume mountPath: /javaheapdump
This allowed heap dumps to be stored temporarily on the worker node whenever an OutOfMemoryError occurred.
Since these files were only required for short-term troubleshooting, persistent storage wasn't necessary.
The Problem
- A few days later, we noticed something unusual.
- Cluster Autoscaler was not removing underutilized worker nodes.
- The node still had available CPU and memory.
- There was no scheduling issue.
- There were no application failures.
- But the EC2 instance remained running.
- Looking at the Cluster Autoscaler logs showed messages similar to:
pod with local storage present
- This was the key clue.
Why This Happens
An emptyDir volume is temporary, but it still uses the worker node's local storage.
When Cluster Autoscaler evaluates whether a node can be removed, it checks whether deleting that node could cause data loss.
Since deleting the node would also delete the contents of the emptyDir volume, Cluster Autoscaler skips the node by default.
This is expected behavior and exists to protect workloads that rely on local storage.
How This Impacts Amazon EKS
In Amazon EKS, worker nodes are usually part of an Amazon EC2 Managed Node Group.
When demand decreases, Cluster Autoscaler attempts to remove underutilized nodes.
A typical scale-down process looks like this:
Underutilized Node │ ▼ Move Application Pods │ ▼ Drain Node │ ▼ Terminate EC2 Instance
However, if a pod is using an emptyDir volume, the process changes:
Underutilized Node │ ▼ Pod Uses Local Storage │ ▼ Cluster Autoscaler Skips the Node │ ▼ EC2 Instance Continues Running
The application remains healthy, but unused EC2 instances continue running, resulting in unnecessary infrastructure costs.
Common Solution
- Many engineers solve this by changing the Cluster Autoscaler configuration:
--skip-nodes-with-local-storage=false
- Although this works, it changes the behavior for the entire Kubernetes cluster.
- Every application using local storage becomes eligible for eviction.
- In a production environment shared by multiple teams, this may not be desirable.
Our Approach
Instead of changing Cluster Autoscaler globally, we made the change only for the affected application.
We added the following annotation to the application's Deployment.
podAnnotations: cluster-autoscaler.kubernetes.io/safe-to-evict-local-volumes: "javaheapdump-volume"
Using Helm, the annotation was added through values.yaml.
podAnnotations: cluster-autoscaler.kubernetes.io/safe-to-evict-local-volumes: "javaheapdump-volume"
The Deployment template simply referenced the annotations.
metadata: annotations: {{- toYaml .Values.podAnnotations | nindent 8 }}
This allowed us to manage the behavior at the application level without modifying Cluster Autoscaler.
Why Application-Level Configuration Is Better
This was the biggest lesson we learned.
- Only our Java application generated temporary heap dumps.
- Other applications in the cluster also used local storage for different purposes.
- Changing Cluster Autoscaler globally would have affected every workload.
- By using an application-specific annotation, we achieved much better control.
Benefits
- Only the required application is affected.
- No changes to the Cluster Autoscaler deployment.
- Other applications using emptyDir continue to be protected.
- Lower risk of accidental data loss.
- Easier to manage through Helm values.
- Different applications can have different eviction policies.
- Safer for shared Amazon EKS clusters.
- No need to restart or reconfigure Cluster Autoscaler.
This approach follows the principle of making the smallest possible change while solving the problem.
Verifying the Configuration
After deploying the updated Helm chart, we verified that the annotation was applied.
kubectl get pod <pod-name> -o yaml
The output showed:
annotations: cluster-autoscaler.kubernetes.io/safe-to-evict-local-volumes: javaheapdump-volume
Once the annotation was present, Cluster Autoscaler could safely evict the pod when scaling down underutilized nodes.
When Should You Use This?
This approach works well when local storage contains temporary or disposable data such as:
- Java heap dumps
- Temporary log files
- Cache files
- Diagnostic output
- Temporary processing data
When Should You Avoid It?
Do not use this annotation if the application stores important business data in local storage.
Examples include:
- Databases
- Uploaded user files
- Persistent application data
- Any data that must survive pod eviction
In those situations, consider using persistent storage such as:
- Amazon EBS with the Amazon EBS CSI Driver
- Amazon EFS
- Amazon S3 for long-term storage of generated files
Best Practices
- Use emptyDir only for temporary data.
- Keep application-specific eviction behavior inside the application, not in Cluster Autoscaler.
- Manage annotations through Helm values to simplify maintenance.
- Test node scale-down behavior in a non-production environment.
- Monitor Cluster Autoscaler logs when troubleshooting scale-down issues.
- Store important diagnostic files in Amazon S3 or persistent volumes if they must be retained.
Conclusion
At first, this looked like a scaling issue.
In reality, Cluster Autoscaler was behaving exactly as designed by protecting pods that used local storage.
Rather than changing the Cluster Autoscaler configuration for the entire cluster, we chose a more targeted solution by adding the cluster-autoscaler.kubernetes.io/safe-to-evict-local-volumes annotation only to the application that generated temporary heap dumps.
This small change allowed underutilized EC2 worker nodes in our Amazon EKS cluster to scale down correctly while keeping the behavior isolated to a single workload.
For production environments, application-level configuration provides better control, reduces operational risk, and avoids unintended side effects on other workloads. It is a simple but effective approach that helped us optimize costs without compromising the stability of the rest of the cluster.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article