Understanding and increasing pod-per-node limits on Amazon SageMaker HyperPod EKS

A deep dive into HyperPod's per-node pod limit and how to increase pod density using DaemonSet scoping and VPC CNI prefix delegation, with notes on caveats when using Karpenter and subnet fragmentation on multi-card EFA instances.

Nicolas

Sr. Specialist SA - AI/ML

Published May 26, 2026

Last modified Jun 9, 2026

Authors: Nicolas Jourdan (Sr. Specialist SA - AI/ML), Giuseppe Angelo Porcelli (Pr. Specialist SA - AI/ML)

Introduction

Amazon SageMaker HyperPod provides resilient, managed clusters for large-scale machine learning training and inference. When orchestrated with Amazon Elastic Kubernetes Service (EKS), HyperPod nodes have a per-instance-type maximum number of pods per node that can be increased to accommodate use cases that require higher pod density on smaller instance types.
This post explains how the pod limit on HyperPod is determined and covers two approaches to increase effective pod density.

The pod-per-node limit is most commonly encountered on small instance types, where the per-node maxPods budget is lowest. For example, ml.g5.2xlarge and ml.g6.2xlarge are common choices for inference pods or interactive IDEs such as SageMaker Spaces on HyperPod because their vCPU and memory match the workload, but each node has a maxPods of 14 per the HyperPod EKS prerequisites table . The same effect appears on smaller CPU node types used as a "platform" tier for cluster-wide operator and controller pods. A significant part of the budget can potentially be consumed by DaemonSets before any workload pod is scheduled, as described in the "Quantifying the system-pod count" section.

Background: how pod IPs are assigned on SageMaker HyperPod EKS

The Amazon VPC CNI assigns every pod an IP from the VPC. Each HyperPod EKS cluster instance has one Elastic Network Interface (ENI) available for pod networking. The ENI has a pool of private IPv4 addresses drawn from the node's subnet. The VPC CNI:

Reserves the primary IPv4 of the ENI for the ENI itself (it is not assigned to any pod).
Hands out secondary IPv4 addresses from the ENI's pool to pods as they schedule onto the node.

The upper bound on pod density per node — enforced by kubelet as maxPods — is derived from this pool. It is derived as the following. Check How maxPods is determined in the EKS User Guide for more details.

1
maxPods = (1 ENI × (IPv4_per_ENI − 1)) + 2

The +2 accounts for the two host-networked pods that are always present (aws-node and kube-proxy), which do not consume a VPC IP but still count toward maxPods. Note that kubelet's maxPods counter includes every pod scheduled on the node, including pods that run with hostNetwork: true. hostNetwork pods share the node's IP address and do not consume a VPC IP from the CNI pool, but they still count toward the maxPods cap.

Every HyperPod instance has one ENI available to the CNI, provisioned at launch, and all pod IPs are assigned to it. On multi-card instances (e.g. ml.p5.48xlarge, ml.trn1.32xlarge), the ENIs on the remaining network cards are reserved for EFA traffic and not available to the CNI.

For ml.g5.2xlarge and ml.m5.2xlarge the kubelet's maxPods limit is 14; for ml.c5.4xlarge it is 29; for ml.p5.48xlarge it is 49. The full per-instance-type table is in the HyperPod EKS prerequisites documentation . The values correspond to IPv4_per_ENI − 1 for each instance type (instance-type dependent; see IP addresses per network interface per instance type ). The maxPods budget therefore maps to a single ENI's address pool minus the primary IP reserved for the ENI itself. You can confirm the value on a running HyperPod node:

1
kubectl get node <node> -o jsonpath='{.status.capacity.pods}'

The value is written by nodeadm into /etc/kubernetes/kubelet/config.json.d/40-nodeadm.conf:

1
2
3
4
5
{
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  "kind": "KubeletConfiguration",
  "maxPods": 14
}

Symptoms

When maxPods is the binding constraint, kubelet rejects additional pod admissions and emits an OutOfpods event:

1
Node didn't have enough resource: pods, requested: 1, used: N, capacity: N

N is the node's maxPods value. At that point, scheduling additional workload pods onto the node requires either freeing existing pods or scaling out to more nodes.

Quantifying the system-pod count

On a HyperPod GPU node, the following DaemonSets are typically present:

kube-system/aws-node (VPC CNI) — hostNetwork
kube-system/kube-proxy — hostNetwork
kube-system/eks-pod-identity-agent — hostNetwork
kube-system/ebs-csi-node, efs-csi-node, fsx-csi-node — depending on installed add-ons
kube-system/nvidia-device-plugin-daemonset, nvidia-container-toolkit-daemonset, nvidia-operator-validator, nvidia-cuda-validator, gpu-feature-discovery
kube-system/dependencies-node-feature-discovery-worker
aws-hyperpod/health-monitoring-agent
hyperpod-observability/hyp-obs-dcgm-exporter, hyp-obs-node-exporter, hyperpod-observability-node-collector

Total: can reach up to ~17 pods depending on which optional add-ons are installed. On an ml.g5.2xlarge node with maxPods=14, the system-pod count alone can reach the cap before any workload pod is scheduled.

Two approaches to mitigate

The rest of this post covers two ways to get around the limit. They are complementary and can be combined.

Approach A: Reduce DaemonSet footprint

Two concrete actions:

1. Delete add-ons you do not use. If your workloads do not mount EFS or FSx, remove the corresponding node DaemonSets permanently by deleting the managed add-ons:

1
2
aws eks delete-addon --cluster-name <cluster> --addon-name aws-efs-csi-driver
aws eks delete-addon --cluster-name <cluster> --addon-name aws-fsx-csi-driver

2. Patch optional DaemonSets to restrict them to nodes that need them. Label nodes by role and update each DaemonSet with a matching nodeSelector or nodeAffinity (see Assigning Pods to Nodes for the available mechanisms: labels, affinity, taints, and tolerations). If the DaemonSet declares a blanket toleration (tolerations: [{operator: Exists}]), clear it so the nodeSelector actually takes effect. For example, to restrict the FSx CSI node driver to nodes that mount FSx:

1
2
3
4
5
6
7
8
kubectl label node <node-name> storage/fsx=true

kubectl patch ds -n kube-system fsx-csi-node --type=strategic -p '{
  "spec":{"template":{"spec":{
    "nodeSelector":{"storage/fsx":"true"},
    "tolerations":[]
  }}}
}'

If a DaemonSet does not need to run on HyperPod nodes at all, exclude them via the sagemaker.amazonaws.com/compute-type=hyperpod label that HyperPod automatically applies to every node it provisions:

1
2
3
4
5
6
7
8
9
10
11
12

kubectl patch ds -n <namespace> <ds-name> --type=strategic -p '{
  "spec":{"template":{"spec":{
    "affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{
      "nodeSelectorTerms":[
        {"matchExpressions":[{"key":"sagemaker.amazonaws.com/compute-type","operator":"NotIn","values":["hyperpod"]}]},
        {"matchExpressions":[{"key":"sagemaker.amazonaws.com/compute-type","operator":"DoesNotExist"}]}
      ]
    }}},
    "tolerations":[]
  }}}
}'

The two nodeSelectorTerms are OR'd: the first matches nodes where the label exists with a value other than hyperpod (other compute types), the second matches nodes where the label is absent altogether (e.g., standard EKS EC2 nodes that aren't HyperPod-managed). Together they cover everything except HyperPod nodes.

kubectl patch does not persist across upgrades: the next time the owning helm chart or operator reconciles the DaemonSet, the spec reverts. For infrequent upgrades this is acceptable — reapply the patches after each upgrade.

Do not remove health-monitoring-agent DaemonSet. This DaemonSet implements HyperPod's node auto-recovery. Per the HyperPod EKS prerequisites , it must be installed for node auto-recovery to work. Leave it on every HyperPod node.

Approach B: Enable VPC CNI prefix delegation (PD) and override the kubelet configuration on HyperPod EKS

When you need high pod density on HyperPod nodes themselves — for example, inference fleets with 50-100 lightweight pods per GPU node — the durable answer is to enable prefix delegation and explicitly override kubelet's maxPods.

How prefix delegation works. With the env var ENABLE_PREFIX_DELEGATION=true set on the aws-node DaemonSet, VPC CNI no longer asks EC2 to attach individual secondary IPs to an ENI. Instead, it attaches IPv4 /28 prefixes: each "slot" on the ENI that used to hold one secondary IP now represents a block of 16 contiguous addresses. The upper bound becomes:

1
(max_ENIs × (IPv4_per_ENI − 1) × 16) + 2

This is clamped to 110 per Kubernetes scalability guidance and the EKS scalability best-practices . See the AWS doc on assigning more IPs to EKS nodes with prefixes for the enablement procedure.

How prefix delegation interacts with HyperPod. On HyperPod, maxPods is set by nodeadm to the single-ENI value and is not recomputed when VPC CNI's prefix-delegation setting changes. To raise it, you write a kubelet configuration drop-in from the HyperPod lifecycle script so kubelet picks it up on first start.

Step 1: Override maxPods in the HyperPod lifecycle script.
In on_create_main.sh (within the S3 bucket referenced by LifeCycleConfig.SourceS3Uri), append:

1
2
3
4
5
6
7
8
9
10
logger "Increasing kubelet maxPods configuration."

mkdir -p /etc/kubernetes/kubelet/config.json.d
cat <<EOF > /etc/kubernetes/kubelet/config.json.d/50-maxpods.conf
{
    "apiVersion": "kubelet.config.k8s.io/v1beta1",
    "kind": "KubeletConfiguration",
    "maxPods": 110 # For very small instance types (t3.medium and below), lower this value — see note below.
}
EOF

The kubelet --config-dir flag loads drop-in files in lexical order, so 50-maxpods.conf overrides the maxPods value in 40-nodeadm.conf. The drop-in is read on kubelet's first start, so no restart is needed when the script runs before kubelet initializes (the default ordering).
110 is the upper bound recommended by Kubernetes scalability guidance and is safe for instance types where the CNI can supply at least that many IPs in single-ENI prefix-delegation mode — t3.large and any g/p/c/r/m/i size from large up. On instance types with fewer than 8 IPv4 addresses per ENI (e.g., t3.medium and below), 110 exceeds the available IP supply; set the drop-in to (IPv4_per_ENI - 1) × 16 + 2 instead, which is 82 on t3.medium, 50 on t3.small, 18 on t3.nano/t3.micro.

Step 2: Enable prefix delegation on the vpc-cni add-on.
The cleanest mechanism is to update the EKS-managed add-on configuration, which persists across add-on version updates:

1
2
3
4
5
6
7
8
9
aws eks update-addon \
 --cluster-name <cluster> \
 --addon-name vpc-cni \
 --configuration-values '{"env":{
 "ENABLE_PREFIX_DELEGATION":"true",
 "MAX_ENI":"1",
 "WARM_ENI_TARGET":"0"
 }}' \
 --resolve-conflicts OVERWRITE

Alternatively, the same env vars can be set directly on the aws-node DaemonSet using the kubectl commands shown in the AWS EKS PD enablement doc :

1
2
3
kubectl set env ds aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true
kubectl set env ds aws-node -n kube-system MAX_ENI=1
kubectl set env ds aws-node -n kube-system WARM_ENI_TARGET=0

These changes do not persist across add-on reconciliations: any subsequent aws eks update-addon or add-on version upgrade will overwrite them. The update-addon --configuration-values form above is the durable path.

MAX_ENI=1 and WARM_ENI_TARGET=0 reflect the fact that HyperPod nodes have only one ENI available for pod networking (described earlier): there is no second ENI for VPC CNI to attach or keep warm.

Effect on existing nodes. Prefix delegation is not retroactive. Nodes that already have secondary IPs attached to their ENIs continue using those allocations; the new setting only applies to ENI operations that happen after it is enabled. Likewise, the kubelet maxPods drop-in from Step 1 is read on first kubelet start, so existing nodes keep their original maxPods=14 (or whatever the single-ENI value was) until they are replaced.
To bring an existing fleet onto prefix-based allocation deterministically, cycle the nodes after enabling PD: cordon, drain, and replace each node so it relaunches through the updated lifecycle script and starts with prefix-mode ENIs from boot.

Scope of this change. Prefix delegation is a cluster-level VPC CNI setting. When enabled, new ENIs start using /28 prefixes for IP allocation. Existing nodes keep their current secondary IPs; only subsequent ENI operations use the new mode.

Note on Karpenter Node Auto-Scaling: Karpenter's scheduler estimates the pod capacity of not-yet-launched nodes assuming a single ENI without Prefix Delegation, not from the maxPods value the actual node will report once it is up. For, e.g., ml.g5.2xlarge / ml.g6.2xlarge that estimate stays around 14 regardless of the lifecycle-script override that raises the running node's maxPods. The practical effect is that during burst scaling, Karpenter initially creates more NodeClaims than required; Setting disruption.consolidationPolicy: WhenEmptyOrUnderutilized with a short consolidateAfter on the HyperPod NodePool lets Karpenter reclaim the under-utilized nodes within a few minutes of a burst stabilizing. If the DaemonSet count that would schedule on a new node exceeds Karpenter's predicted maxPods, the scheduler simulation finds no room for the pending pod and does not create a NodeClaim. Approach A (reducing the DaemonSet footprint) brings the simulated count below the predicted maxPods and is therefore a prerequisite for Karpenter provisioning on small instance types where DaemonSets dominate the budget.

The subnet fragmentation problem on large EFA instances

Prefix delegation introduces a subnet-level precondition that matters specifically on HyperPod instances with many EFA network interfaces. When using prefix delegation, EC2 allocates each /28 prefix from a contiguous block of 16 IPs within the subnet. If the subnet has many individual IPs scattered across /28 boundaries — regardless of whether the IPs are currently in use — EC2 cannot satisfy prefix requests and returns InsufficientCidrBlocks. This is documented in the AWS EKS PD enablement procedure and described in detail in the Prefix Mode for Linux best-practices page.

On HyperPod, every network card on a multi-card instance gets its own ENI with a primary IPv4 address. Those primary IPs are allocated from the subnet CIDR and are not clustered — EC2 draws them from the general pool, which scatters them across /28 boundaries as the subnet fills. The math on ml.p5.48xlarge for example becomes:

Per instance: 32 ENIs, each consuming one primary IP.
Each primary IP sits in a different /28 block (worst case).
One instance can contaminate up to 32 distinct /28 blocks in its subnet.

Depending on the subnet size, a small number of multi-card instances can consume most or all of the /28 boundaries the subnet can form, even when absolute IP usage looks low. At that point, prefix allocations begin to fail with InsufficientCidrBlocks even though the subnet still has many free individual addresses. There are two ways to mitigate this issue:

Mitigation Option 1: Launch EFA interfaces as `efa-only`.

Instance groups on HyperPod EKS accept a NetworkInterface.InterfaceType field that controls how the EFA interfaces on cards 1 through N are provisioned. Setting it to efa-only (instead of the default efa) makes those interfaces EFA-only — they have no IPv4 address and consume nothing from the subnet. Only the primary network interface on card 0 stays EFA-with-ENA, because the primary network interface must be ENA-capable. This produces the AWS-recommended "Save IP addresses" topology described in EFA configuration for P5 and P5e instances .

On a multi-card instance such as ml.p5.48xlarge, this drops the number of primary IPs allocated per instance from 32 (one per card, scattered across /28 boundaries) down to 1 (only on card 0). The pool of IPs that VPC CNI uses for pod networking still lives on card 0 and is not affected. Combined with prefix delegation, the on-card-0 pool is allocated as /28 prefixes that fit cleanly inside a subnet CIDR reservation, and the per-card-N scattering disappears at the source.

The NetworkInterface.InterfaceType field can be set both at cluster creation time via CreateCluster and on existing clusters via UpdateCluster — for existing instance groups, add the field to the IG spec the same way you would change any other property. The setting only affects newly provisioned instances; instances that were already launched keep their current ENI topology until they are replaced.

1
2
3
4
5
6
7
8
9
{
  "InstanceGroupName": "auto-spot-p5-48xlarge", # Example
  "InstanceType": "ml.p5.48xlarge",
  "InstanceCount": 1,
  ...
  "NetworkInterface": {
    "InterfaceType": "efa-only"
  }
}

Mitigation Option 2: Subnet CIDR reservations

The AWS-supported mitigation is a subnet CIDR reservation of type prefix :

1
2
3
4
aws ec2 create-subnet-cidr-reservation \
  --subnet-id <subnet> \
  --reservation-type prefix \
  --cidr <contiguous-block-inside-the-subnet>

After the reservation is created:

EC2 will not allocate individual IPs (primary or secondary) inside the reserved range.
EC2 will allocate /28 prefixes (for VPC CNI's PD) inside the reserved range when asked.
Existing IPs already within the range are retained, but EC2 will not re-issue those addresses as individual IPs once they are released.

The practical effect: the reserved range becomes a "clean room" of contiguous /28 blocks that VPC CNI's Prefix Delegation allocations can draw from, while EFA primary IPs and other individual allocations are confined to the unreserved portion of the subnet.

Sizing. A reserved /22 (1,024 IPs, 64 /28 blocks) supports approximately 64 ÷ ceil(110/16) ≈ 9 concurrent HyperPod nodes at maxPods=110 each. For larger fleets, reserve a larger block or add additional reservations. Keep at least half of the parent subnet unreserved so EFA ENI primary IPs and other non-Prefix Delegation allocations have somewhere to go.

Order of operations. Create the reservation before enabling prefix delegation. If Prefix Delegation is enabled first, VPC CNI may allocate /28 prefixes in scattered locations throughout the subnet; those allocations do not move into the reservation once it exists. The same concern applies to existing nodes: because Prefix Delegation does not affect already-attached secondary IPs, any node that was running before Prefix Delegation was enabled will keep allocating from its original (scattered) IP pool until it is replaced. Create the reservation, enable Prefix Delegation, then cycle the existing nodes so replacement nodes draw /28 prefixes from the clean reserved range from their first ENI onward.

Multi-subnet considerations. Subnet CIDR reservations are per-subnet, so create a reservation in each subnet that nodes might land in. The exact behavior differs depending on the type of instance group: Standard instance groups will only utilize a single subnet for IP addresses even if multiple subnets are configured for that instance group. Flexible instance groups with multiple subnets will instead distribute instances across the configured subnets rather pin all to one. This is a useful way to soften per-subnet fragmentation: EFA primary IPs spread across multiple subnets instead of concentrating in one, so each subnet's PD-usable /28 capacity is less affected.

Summary and best practices

On HyperPod, kubelet's maxPods ceiling maps to a single ENI's IP pool: roughly IPv4_per_ENI − 1. For ml.g5.2xlarge / ml.g6.2xlarge this is 14; for ml.c5.4xlarge it is 29; for ml.p5.48xlarge it is 49.
hostNetwork pods do not consume a VPC IP but still count toward maxPods.
System and DaemonSet pods on a fresh HyperPod node can reach up to ~17 in total, consuming a large share of the cap.
If you use Karpenter, do Approach A first: its scheduler will keep using the per-instance-type default maxPods (not the lifecycle-script override), and a DaemonSet pod count above this value will block new-node provisioning entirely, independent of Prefix Delegation.

To increase effective pod density:

Reduce what lands on the node. Delete unused managed add-ons (EFS/FSx CSI). Patch remaining optional DaemonSets with a nodeSelector and remove blanket tolerations so they only run where needed. Do not remove health-monitoring-agent.
Enable prefix delegation and override kubelet via lifecycle scripts for high-density HyperPod workloads. Prefix delegation is not retroactive — existing nodes keep their current secondary IPs and original maxPods until replaced, so plan a node cycle as part of the rollout.

If your HyperPod fleet uses instances with many EFA network cards (p5, p6, trn2), plan for subnet fragmentation from EFA primary IPs by either setting the relevant instance groups network interface configuration to efa-only or creating subnet CIDR reservations of type prefix sized for your expected pod density, before enabling PD.

# amazon-sagemaker-hyperpod # kubernetes # amazon-eks # ai # ai-ml

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Understanding and increasing pod-per-node limits on Amazon SageMaker HyperPod EKS

Nicolas

Introduction

Background: how pod IPs are assigned on SageMaker HyperPod EKS

Symptoms

Quantifying the system-pod count

Two approaches to mitigate

Approach A: Reduce DaemonSet footprint

Approach B: Enable VPC CNI prefix delegation (PD) and override the kubelet configuration on HyperPod EKS

The subnet fragmentation problem on large EFA instances

Mitigation Option 1: Launch EFA interfaces as efa-only.

Mitigation Option 2: Subnet CIDR reservations

Summary and best practices

Mitigation Option 1: Launch EFA interfaces as `efa-only`.