How to Enable Swap Memory on EKS Using Karpenter
Learn how to configure swap memory on Amazon EKS using Karpenter, so your pods survive memory spikes without getting OOMKilled — no over-provisioning required.
For years, Kubernetes engineers have accepted this reality as the price of predictable resource management. The conventional wisdom? Keep memory requests and limits identical, over provision your nodes, and hope for the best. But what if there's a more nuanced approach?
The Memory Management Challenge in Cloud Environments
Traditional data centers and cloud platforms handle memory differently. In a traditional environment, administrators could rely on swap as a safety valve—a pressure release mechanism that sacrificed performance for availability. When RAM filled up, the kernel would gracefully move inactive pages to disk, buying time to respond.Kubernetes intentionally broke from this model. The platform's architects prioritized predictability over flexibility, arguing that swap introduced too much variability in multi-tenant clusters. For years, the kubelet wouldn't even start if it detected swap on a node. This decision forced teams into a corner: either over provision memory significantly or accept that memory spikes would kill pods. The cost implications are substantial. Multiply that across dozens or hundreds of nodes, with 20-30% over provisioning for safety, and you're looking at tens of thousands of dollars in unnecessary cloud spend annually.
What is Swap?
Swap is a memory management mechanism where disk space serves as extra RAM. When your system runs low on physical memory, it transfers inactive data from RAM to the swap area on disk. This frees up RAM for currently active processes. Essentially, swap acts as a safety net that helps systems continue operating even when memory demands exceed available RAM.
How Swap Works in Modern Kubernetes
When enabled, swap integrates with cgroups v2 to provide controlled access to secondary storage. The key innovation is the LimitedSwap behavior mode, which restricts containers to using memory plus swap up to their configured memory limit. This maintains resource isolation while providing flexibility during transient spikes.Think of it as a shock absorber for memory pressure. When a pod's working set temporarily exceeds its baseline usage, the kernel can move cold pages to swap rather than immediately triggering the OOM killer. This buys crucial seconds—or minutes—for autoscaling to provision additional capacity or for the workload to complete its current task.
When Swap Becomes Your Strategic Advantage
Swap isn't a universal solution, but in the right contexts, it's transformative. Let's examine scenarios where swap delivers real value:
Scenario 1: JVM-Based Applications with Large Heaps
Java applications typically allocate substantial heap space at startup—often several gigabytes—but only actively use a fraction of that memory during normal operation. Consider a microservice running Spring Boot with a 4GB heap that typically uses 1.5GB. Without swap, you're forced to provision 4GB of RAM. With swap on fast NVMe storage, you can provision 2GB of RAM plus 2GB of swap, reducing node costs while maintaining application stability.
Scenario 2: Batch Processing with Predictable Spikes
Data processing pipelines often experience periodic memory spikes during operations like sorting, aggregation, or report generation. These spikes are brief but intense. Swap allows you to size nodes for typical workload demands while gracefully handling peak usage without pod evictions
The Performance-Reliability Trade-off
Let's be direct: swap is slower than RAM. How much slower depends on your storage:
- NVMe SSD: ~100x slower than RAM (still sub-millisecond latency for many operations)
- SATA SSD: ~500x slower than RAM
- Network-attached EBS volumes: ~1000x+ slower than RAM with additional network latency
This performance differential creates three critical implications you must consider:
1. Unpredictable Latency Patterns
When your application begins swapping, request latencies can spike dramatically. A typical API response that takes 50ms might suddenly take 500ms or more. For user-facing applications with strict SLA requirements, this variability is often unacceptable. Swap works best for workloads that can tolerate occasional performance degradation.
2. Security Considerations
Swap persists memory contents to disk, potentially exposing sensitive data like API keys, session tokens, or customer information. In regulated industries (finance, healthcare), unencrypted swap space may violate compliance requirements. Always encrypt swap when handling sensitive workloads.
3. The Noisy Neighbor Problem
In multi-tenant clusters, one pod aggressively using swap can degrade I/O performance for other workloads on the same node. This is why LimitedSwap is crucial—it prevents runaway containers from monopolizing swap resources.
The Journey of Swap Support in Kubernetes
- Kubernetes 1.22 (August 2021): Alpha support for swap was introduced. This was the first experimental implementation, allowing users to test swap functionality but with significant limitations and bugs.
- Kubernetes 1.28 (August 2023): Swap support graduated to Beta. This was a major milestone - the feature became more stable, added cgroup v2 support, introduced LimitedSwap behavior, and included comprehensive testing.
- Kubernetes 1.32 (December 2024): Significant improvements and refinements were made. Enhanced stability features, better security (memory-backed volumes protection), removal of UnlimitedSwap behavior, and high-priority pod protections were added.
For Amazon EKS users:
- EKS 1.28+: You can use swap with Beta stability. This is the minimum version we recommend in the blog post.
- EKS 1.32+: You get the latest improvements including better security protections and stability enhancements.
Architecture Summary
Implementing Swap in Amazon EKS: A Practical Guide
Prerequisites and Planning
Before implementation, verify your environment meets these requirements:
- Amazon EKS version 1.28 or later (will sure 1.34 in this )
- Karpenter version 0.37+ for automated node provisioning
- Amazon Linux 2023 (AL2023) node AMI—AL2 lacks cgroupsv2 support
- Instance types with NVMe storage (C5d, M5d, R5d families) for optimal performance
Step 1: Create a Karpenter NodeClass and NodePool
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
# =============================================================
# 01-ec2nodeclass-swap.yaml — Swap-enabled EC2NodeClass
# =============================================================
# Same as "default", but the userData block:
# 1. Creates a 10 GB swap file on NVMe instance store
# (falls back to root EBS if no instance store).
# 2. Configures kubelet: failSwapOn=false, NodeSwap feature
# gate ON, swapBehavior=LimitedSwap.
# =============================================================
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
name: swap-enabled
annotations:
kubernetes.io/description: >
Swap-enabled node class. Creates a 10 GB swap file and
tells kubelet to allow LimitedSwap.
spec:
role: KarpenterNodeRole-swap-eks
amiSelectorTerms:
- alias: al2023@latest
subnetSelectorTerms:
- tags:
karpenter.sh/discovery: swap-eks
securityGroupSelectorTerms:
- tags:
karpenter.sh/discovery: swap-eks
instanceStorePolicy: RAID0
blockDeviceMappings:
- deviceName: /dev/xvda
ebs:
volumeType: gp3
volumeSize: 50Gi
encrypted: true
deleteOnTermination: true
# -------------------------------------------------------------
# userData — multipart MIME document understood by AL2023.
#
# Part 1 : shell script → creates & enables the swap file
# Part 2 : NodeConfig → kubelet flags for swap support
# -------------------------------------------------------------
userData: |
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"
--//
Content-Type: text/x-shellscript; charset="us-ascii"
#!/usr/bin/env bash
set -euo pipefail
# Use NVMe instance store when available (fastest),
# otherwise fall back to the root EBS volume.
if mountpoint -q /mnt/k8s-disks/0; then
SWAPFILE="/mnt/k8s-disks/0/swapfile"
else
SWAPFILE="/swapfile"
fi
echo "Creating 10 GB swap file at: ${SWAPFILE}"
sudo fallocate -l 10G "${SWAPFILE}"
sudo chmod 600 "${SWAPFILE}"
sudo mkswap "${SWAPFILE}"
sudo swapon "${SWAPFILE}"
echo "Swap enabled. Verify with: free -h"
--//
Content-Type: application/node.eks.aws
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
kubelet:
config:
failSwapOn: false # allow nodes with swap to join
featureGates:
NodeSwap: true # enable the swap feature gate
memorySwap:
swapBehavior: LimitedSwap # containers only use swap up to (limit - request)
--//1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
# =============================================================
# 02-nodepool-swap.yaml — Swap-enabled NodePool
# =============================================================
# Routes workloads that need swap onto nodes created by the
# "swap-enabled" EC2NodeClass. A NoSchedule taint keeps
# unintended pods off these nodes.
# =============================================================
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: swap
annotations:
kubernetes.io/description: >
Swap pool. Nodes here have a 10 GB swap file and kubelet
configured for LimitedSwap. Only pods that tolerate the
"swap=true" taint will be scheduled here.
spec:
template:
metadata:
labels:
pool: swap
spec:
# Reference the swap-enabled node class
nodeClassRef:
group: karpenter.k8s.aws
kind: EC2NodeClass
name: swap-enabled
# Taint — pods MUST add a matching toleration:
# tolerations:
# - key: swap
# value: "true"
# effect: NoSchedule
taints:
- key: swap
value: "true"
effect: NoSchedule
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["spot", "on-demand"]
- key: kubernetes.io/arch
operator: In
values: ["amd64"]
- key: kubernetes.io/os
operator: In
values: ["linux"]
# Prefer instance families that ship with NVMe instance
# storage (m, c, r) for fastest swap performance.
- key: karpenter.k8s.aws/instance-category
operator: In
values: ["m", "c", "r"]
- key: karpenter.k8s.aws/instance-generation
operator: Gt
values: ["2"]
# Minimum 2 GB RAM so the 10 GB swap file is meaningful
- key: karpenter.k8s.aws/instance-memory
operator: Gt
values: ["2047"] # value is in MiB
limits:
cpu: "100"
memory: "400Gi"
disruption:
consolidationPolicy: WhenEmptyOrUnderutilized
consolidateAfter: 5m
Step 2: Grafana Dashboard for Swap Metrics
1
2
3
4
5
6
7
8
9
10
11
12
promql
# Swap Total per node
node_memory_SwapTotal_bytes
# Swap Used per node
node_memory_SwapTotal_bytes - node_memory_SwapFree_bytes
# Swap Free per node
node_memory_SwapFree_bytes
# Swap Usage Percentage
(1 - (node_memory_SwapFree_bytes / node_memory_SwapTotal_bytes)) * 100- Open Grafana → Click "+" → "New dashboard"
- Click "Add visualization"
- Select Prometheus data source
- Enter query (e.g.,
node_memory_SwapTotal_bytes) - Set Legend:
{{instance}} - Set Unit: Data → bytes(IEC)
- Click Apply
- Save dashboard
Step 3: Deploying Test job
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
# =============================================================
# 03-test-jobs.yaml — Swap validation test jobs
# =============================================================
# # Prerequisites:
# - EKS cluster with Kubernetes 1.34+
# - Karpenter installed with swap-enabled NodePool
# - EC2NodeClass configured with swap userData
#
# ============================================================================
# JOB DESCRIPTIONS
# ============================================================================
#
# 1. memory-burst-no-swap (Guaranteed QoS)
# ─────────────────────────────────────
# - Memory Request: 512Mi
# - Memory Limit: 512Mi (request == limit)
# - Allocation: Attempts to allocate 1.8GB
# - Expected: OOMKilled
# - Why: With LimitedSwap policy, Guaranteed QoS pods (request == limit)
# have NO access to swap. When memory exceeds the limit, the
# container is terminated by the OOM killer.
#
# 2. memory-burst-with-swap (Burstable QoS)
# ──────────────────────────────────────
# - Memory Request: 512Mi
# - Memory Limit: 2560Mi (request < limit)
# - Allocation: Attempts to allocate 1.8GB
# - Expected: Completed successfully
# - Why: Burstable QoS pods (request < limit) can use swap up to
# (limit - request). This pod can spill ~2GB into swap,
# allowing the 1.8GB allocation to succeed.
#
# 3. swap-usage-monitor (Monitoring)
# ────────────────────────────────
# - Runs alongside the stress tests
# - Reads /proc/meminfo every 5 seconds for 50 seconds
# - Outputs: MemTotal, MemFree, MemAvailable, SwapTotal, SwapFree
# - Purpose: Observe real-time swap usage during memory pressure
#
# ============================================================================
# USAGE
# ============================================================================
#
# Apply jobs:
# kubectl apply -f 06-test-jobs.yaml
#
# Watch pod status:
# kubectl get pods -l test-suite=swap-validation -w
#
# View swap monitor logs:
# kubectl logs -l scenario=monitoring -f
#
# Check job results:
# kubectl get jobs -l test-suite=swap-validation
#
# Cleanup:
# kubectl delete -f 06-test-jobs.yaml
#
# ============================================================================
apiVersion: batch/v1
kind: Job
metadata:
name: memory-burst-no-swap
labels:
test-suite: swap-validation
scenario: guaranteed-qos
annotations:
description: "Guaranteed QoS pod - no swap access, expects OOMKill"
spec:
ttlSecondsAfterFinished: 300
backoffLimit: 0
template:
metadata:
labels:
test-suite: swap-validation
scenario: guaranteed-qos
spec:
nodeSelector:
pool: swap
tolerations:
- key: swap
value: "true"
effect: NoSchedule
terminationGracePeriodSeconds: 5
containers:
- name: memory-allocator
image: polinux/stress
command: ["stress"]
args:
- "--vm"
- "1"
- "--vm-bytes"
- "1800M"
- "--vm-hang"
- "0"
- "--timeout"
- "45s"
resources:
requests:
memory: "512Mi"
cpu: "100m"
limits:
memory: "512Mi"
cpu: "200m"
restartPolicy: Never
apiVersion: batch/v1
kind: Job
metadata:
name: memory-burst-with-swap
labels:
test-suite: swap-validation
scenario: burstable-qos
annotations:
description: "Burstable QoS pod - can use swap, expects completion"
spec:
ttlSecondsAfterFinished: 300
backoffLimit: 0
template:
metadata:
labels:
test-suite: swap-validation
scenario: burstable-qos
spec:
nodeSelector:
pool: swap
tolerations:
- key: swap
value: "true"
effect: NoSchedule
terminationGracePeriodSeconds: 5
containers:
- name: memory-allocator
image: polinux/stress
command: ["stress"]
args:
- "--vm"
- "1"
- "--vm-bytes"
- "1800M"
- "--vm-hang"
- "0"
- "--timeout"
- "45s"
resources:
requests:
memory: "512Mi"
cpu: "100m"
limits:
memory: "2560Mi"
cpu: "200m"
restartPolicy: Never
apiVersion: batch/v1
kind: Job
metadata:
name: swap-usage-monitor
labels:
test-suite: swap-validation
scenario: monitoring
annotations:
description: "Monitors swap usage during test execution"
spec:
ttlSecondsAfterFinished: 300
backoffLimit: 0
template:
metadata:
labels:
test-suite: swap-validation
scenario: monitoring
spec:
nodeSelector:
pool: swap
tolerations:
- key: swap
value: "true"
effect: NoSchedule
containers:
- name: swap-monitor
image: busybox:1.36
command: ["/bin/sh", "-c"]
args:
- |
echo "=== Swap Monitor Started ==="
echo "Hostname: $(hostname)"
echo ""
for i in $(seq 1 10); do
echo "--- Check $i/10 ---"
cat /proc/meminfo | grep -E "Swap|Mem"
echo ""
sleep 5
done
echo "=== Swap Monitor Complete ==="
resources:
requests:
memory: "32Mi"
cpu: "50m"
limits:
memory: "64Mi"
cpu: "100m"
restartPolicy: Never
Test Results
| Test | QoS Class | Memory Request | Memory Limit | Allocation | Result |
|---|---|---|---|---|---|
| memory-burst-no-swap | Guaranteed | 512Mi | 512Mi | 1.8GB | OOMKilled |
| memory-burst-with-swap | Burstable | 512Mi | 2.5Gi | 1.8GB | Completed |
| swap-usage-monitor | Burstable | 32Mi | 64Mi | - | Completed |
Node Provisioned: m3.medium (spot) - 3.8GB RAM, 10GB swap
The results prove that:
- When
request == limit, containers have no swap access (LimitedSwap policy) - When
request < limit, containers can use swap up to(limit - request)
Conclusion
Swap in Kubernetes represents a maturation of the platform's resource management capabilities. It's not a replacement for proper capacity planning or a justification for under-provisioning infrastructure. Rather, it's a strategic tool for specific use cases where the performance-availability trade-off makes sense.The key insight is that different workloads have different requirements. User-facing APIs serving millions of requests per second need predictable, low-latency responses—swap is inappropriate. Batch processing jobs that run overnight and can tolerate occasional slowdowns? Swap can reduce infrastructure costs by 20-30% without impacting business outcomes
References
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article