Scale-Driven Architecture: Lessons from a Hyperscale Multi-Tenant Agent Platform on EKS — Part 1: Networking and Pod Density
Running 100k+ tenants on a single EKS cluster is not a larger version of running 100 — it's a different problem entirely. This first part walks the capacity layer where the walls show up first: how VPC IP space becomes capacity planning, two fundamentally different paths to expanding the pod IP pool, and the layered ceilings that decide how many pods a node will really take.
GPart 1 — Capacity Before Architecture: Networking and Pod Density
A three-part series. Part 1 of 3.
Running 100k+ tenants on a single EKS cluster is not a larger version of running 100. It is a different problem, with different primitives, and a different set of things that will break first.
This series is about building a hyperscale multi-tenant SaaS platform on EKS on top of OpenClaw, an open-source autonomous AI agent framework. OpenClaw itself is designed for single-user operation — one instance serves one user — which works cleanly for individual use but becomes the central engineering problem the moment you want to offer it as a service to many users at once. In this deployment shape, a "tenant" is an end user of the hosted service, and each tenant gets their own dedicated OpenClaw agent workload — an isolated pod (or pod set) with its own configuration, agent state, and lifecycle — running inside a single shared EKS cluster. The target scale — 100k+ tenants per cluster, which also means 100k+ pods — was not something we grew into. It was the starting constraint. Every architectural choice was reasoned backward from that number, not forward from a smaller system that happened to get popular.
That reversal matters, because it changes which patterns are viable. The optimal design at 1k tenants and the optimal design at 100k tenants are often not the same design at different sizes — they are different designs entirely. A pattern that is clean, idiomatic, and correct at small scale can become structurally impossible past a certain threshold, not because it performs poorly, but because it runs out of quota, or out of address space, or out of something you did not think to count.
At hyperscale, every layer has a limit. Kubernetes itself recommends clusters stay under 5,000 nodes and 150,000 pods — that is the headroom you are working inside, and 100k+ pods already consumes most of it. Underneath that envelope sit the per-layer ceilings this series works through:
In this Part:
- A VPC CIDR caps at /16 (65,536 addresses) — Networking.
Later in the series:
- EFS access points cap at 10k per filesystem — and the EFS CSI Driver hardcoded 1k for a while. Part 2, Storage.
- EKS's Provisioned control plane caps etcd at 16 GB, with K8s scheduling throughput regressions at high PVC counts. Part 3, Scaling.
- EKS Pod Identity associations cap at 5,000 per cluster — a hard, non-adjustable quota. Part 3, IAM.
Plus an unavoidable supporting cast of IAM roles, ENIs, IP prefixes, and volume attachments — all finite, all interacting.
The design question is rarely "what is the best solution" — it is "which limit do we hit first, and what does the layer above look like once we route around it." The feasible design space is whatever is left after these ceilings compose.
This series walks that space bottom-up, in the order the limits actually appeared. Part 1 starts at the capacity layer — networking and pod density — because that is where the walls show up first. Part 2 covers the trust and persistence layers (isolation and storage). Part 3 closes on the composition itself: scaling behavior and IAM, where the quota ceilings stop being independent and start multiplying.
Networking: IP Space Is Capacity Planning, Not Network Engineering
The first wall in a 100k+ tenant cluster is not throughput, latency, or routing policy. It is address space.
EKS with VPC CNI gives every pod a real VPC IP. That is a strong model — pods are first-class network citizens, routable and observable the same way EC2 instances are — but it makes IP consumption linear in pod count. At 100k+ pods, a single VPC CIDR is nowhere near enough. A VPC CIDR is capped at /16, which is 65,536 addresses. A single /16 cannot hold 100k+ pods, and no amount of clever routing changes the arithmetic.
The shape of the problem, then, is how to expand the IP pool available to pods — and this has to be settled before almost any other networking decision. Subnets, once handed out to running pods, are not easy to take back. IP allocation belongs in the same bucket as etcd's 16 GB ceiling from the opener: a capacity constraint that shapes the design space rather than one the design chooses from.
Two Paths to Expanding the Pod IP Pool
There are two fundamentally different ways to expand pod IP capacity on EKS. They are not competing implementations of the same idea — they are different trade-offs against a different scarce resource. Which one fits depends on whether your bottleneck is AWS-side IP space or organization-side routable RFC1918 allocation.
Custom Networking: pods on a non-routable secondary CIDR
EKS Custom Networking lets you place pods on a completely separate, non-routable CIDR — typically from the
100.64.0.0/10 CGNAT range — while nodes stay on the VPC's primary routable range. The mechanism is the ENIConfig custom resource: each (AZ, subnet) combination needs its own ENIConfig, which tells VPC CNI to attach a secondary ENI in the non-routable range and assign pod IPs from there.The value proposition is specific and real: pod IPs no longer consume routable RFC1918 space. At organizations where routable address allocation is tight — VPCs have to coexist with on-prem networks, Direct Connect, peered accounts — this is sometimes the only way to land 100k+ pods without triggering cross-VPC routing conflicts.
The cost is configuration surface. You need one
ENIConfig per (AZ, subnet) combination, and each one has to stay in sync with:- The subnet's actual CIDR and security groups
- Karpenter's
EC2NodeClasssubnet selector, which has to match - Karpenter's
NodePoolAZ constraints, which have to match - Any change to the underlying subnet topology
A single mismatch — a
NodePool scheduling a node into an AZ whose ENIConfig has drifted, or an EC2NodeClass selecting a subnet that no ENIConfig covers — does not produce a clean error. It produces nodes that come up but cannot schedule pods, or pods stuck in ContainerCreating with CNI errors that require cross-referencing three different resources to debug. The configuration surface grows linearly in AZ × subnet, and the coupling with Karpenter is tight enough that any team operating the cluster has to reason about both systems' selectors at once.At a few subnets in one or two AZs, this is manageable. At the subnet count a 100k+ cluster actually needs — multiple AZs, many pod subnets per AZ to spread capacity — it becomes an ongoing source of drift.
Enhanced Subnet Auto Discovery: scaling the routable pool cleanly
If routable RFC1918 space is available, there is a different path entirely. Enhanced Subnet Auto Discovery lets VPC CNI discover pod subnets automatically via a tag (
kubernetes.io/role/cni). Nodes and pods both live in routable space; the pod IP pool is expanded by adding more routable secondary CIDRs to the VPC (default 5 associations, raisable to 50) and tagging the new subnets.There is no
ENIConfig to maintain, no per-subnet CR to keep in lockstep with Karpenter. Adding a new AZ or a new pod subnet is a tag operation, and VPC CNI picks it up. The configuration surface is flat in the dimension that actually grows — subnet count — which is exactly the dimension that makes ENIConfig painful.This is the path this cluster took. It assumes routable address space is not the binding constraint, which is a real precondition to check before committing: if the organization cannot allocate enough RFC1918 for 100k+ pods without triggering routing conflicts, this path is closed and Custom Networking becomes necessary despite its cost.
A working Terraform PoC that stands up both patterns (Custom Networking with
ENIConfig and Enhanced Subnet Auto Discovery) side by side is available at shawnawshk/tf-eks-advanced-networking-demo , for anyone who wants to see the configuration surface of each in code rather than in prose.One Networking Decision That Has to Be Made Before You Need It
There is one more networking-layer decision that belongs here, even though its reason won't be fully explained until the next section: CIDR Reservation on pod subnets.
If you plan to push pod density up with VPC CNI's prefix mode — which you almost certainly will at 100k+ (see the next section) — each pod subnet needs a block reserved for prefix allocation at subnet creation time. The mechanism is
aws ec2 create-subnet-cidr-reservation with --reservation-type prefix, and it prevents secondary IP allocation from fragmenting the address space that prefix mode needs to carve up into aligned /28 blocks.The why — why prefix mode fragments subnets, what the symptom looks like, why it is invisible until it isn't — belongs in the pod density section, and that is where it will be covered. The reason it has to be flagged here is simpler and more operational: this is a subnet-creation-time decision. Once pods are running in a subnet and the address space has been chewed up by normal churn, there is no in-place fix. You cannot reclaim the gaps. Whatever reservation policy you wanted, you needed it at
CreateSubnet.This is the canonical version of this section's running theme: networking decisions live upstream of almost every other choice, and the window to make them cleanly closes fast.
What Networking Taught Us
Two things carry forward into the rest of the series.
First, IP address space is a capacity problem, not a network engineering problem. The /16 cap is a hard edge; at 100k+ you are always multi-CIDR, and the real question is which kind of address space you have to spend. Once pods are living in a subnet, the decision is effectively permanent.
Second, the two paths here solve different scarcity problems, and it matters to say so out loud. Custom Networking buys you relief from routable RFC1918 pressure at the cost of a configuration surface that grows with subnet count. Enhanced Subnet Auto Discovery keeps the configuration surface flat but only works if routable space is available to spend. Framing them as "two implementations of the same feature" would hide the trade-off that actually drives the choice.
Next: from IP capacity to pod capacity — and the layered ceilings that decide how many pods a node will really take.
Pod Density: A Triangle, Not a Target
Once the IP pool is sorted, the next question is how efficiently each node gets used. At 100k+ pods, small differences in pods-per-node translate into large differences in node count, and node count drives almost every downstream cost: EC2 spend, scheduling latency, control-plane load, observability cardinality, operational surface.
The working target for this cluster was ~200+ pods per node. That number is worth unpacking, because it is not the maximum the hardware could support. It is the intersection of three constraints, and pushing past it on any single axis costs you on another.
The three axes:
- Instance family and generation. The cluster runs on Graviton (arm64), broadly across current-generation C / M / R families. This follows from the application side — OpenClaw supports arm64 natively — and Graviton being the economical default once arm64 is in play; it is not itself a density decision.
- Instance network capacity. Per-instance pod ceiling is a product of ENI count × prefixes per ENI × IPs per prefix. This is a hardware-level cap; no amount of CNI tuning moves it, and it varies meaningfully across instance families and sizes. The instance choice and the density target have to be picked together — density is a function of the instance type, not an independent knob.
- Cost per pod. Larger instances give better pod-per-dollar economics on paper, but they also enlarge the blast radius of a single node failure and coarsen scheduling granularity. A cluster made of very large nodes fragments capacity in different ways than one made of many medium nodes, and bin-packing efficiency on mixed workloads degrades.
None of these three is independently adjustable. Raising the density target usually means larger instances, which concentrates risk and reduces scheduling flexibility. Prioritizing smaller nodes for blast-radius reasons caps density and pushes node count — and with it, everything node count drives — up.
There is also a pragmatic constraint sitting on top of the triangle: Kubernetes bin-packing is only as useful as the CPU and memory available per pod. A node that fits 250 pods but only leaves each one with a fraction of a vCPU and a few hundred MB of memory is not running 250 real workloads. The density target has to stay honest about what each pod actually needs — and there is a second, request-driven ceiling that formalizes exactly that, covered after
maxPods below."Maximum density" and "optimal density" are different numbers, and the difference matters. The instance-level ceiling tells you what is possible. The triangle tells you where you actually want to sit inside that ceiling. The landing point for this cluster was 216 pods per node — not a value derived from a closed-form calculation, but the point where the density-vs-cost tension balanced for this workload shape. It is specific enough to be useful as a reference, general enough that anyone reading this should expect their own number to differ.
A Hedge the Triangle Doesn't Show: Instance Diversity
The three axes above frame density as a single-instance-type decision — pick one family and size, optimize within it. At 100k+ scale, that framing is itself a risk. If every node comes from one EC2 family and size, the cluster is also binding its capacity plan to one EC2 pool's spare capacity in each AZ. When that pool tightens — and at scale it will, episodically — node provisioning blocks, and the carefully-chosen density target stops mattering because you can't get the nodes.
Node provisioning in this cluster is driven by Karpenter, whose
NodePool config lets you enumerate a list of eligible instance types rather than a single pick. Karpenter picks the cheapest available option at provisioning time, and falls back to other entries in the list when the preferred pool runs dry. The list covers multiple compatible Graviton sizes across current-generation C / M / R families — picked to give Karpenter enough flexibility that at least one eligible pool is always available, without widening the set so far that pod-per-node arithmetic becomes unpredictable.This is an operational hedge, not a density decision — but it interacts with density in one specific way: the
maxPods analysis below has to hold for every instance type in the list, not just the one you expect to land on most often. Miss that, and a Karpenter fallback to an off-nominal instance type silently changes the pod ceiling per node without anyone noticing until the scheduler runs out of capacity it thought it had.The ceiling above the triangle: how maxPods is actually determined
Before going further, there is one more ceiling that sits above everything the triangle just described: the
maxPods value the kubelet is booted with. And the thing worth knowing — the thing that trips up teams reading eni-max-pods.txt and assuming the number they see there is what they will get — is that the final maxPods is decided by a precedence chain , not a single source. From highest priority to lowest:- Managed Node Group enforcement. If you use a Managed Node Group without a custom AMI, EKS injects
maxPodsinto the user data with a hard cap:110for instances with fewer than 30 vCPUs,250for instances with 30+ vCPUs. This overrides everything below. - Explicit
maxPodsin kubelet config (typically via a launch template with a custom AMI). nodeadm maxPodsExpressioninNodeConfig— only effective if no higher-precedence source has already set the value.- Default: the ENI-based formula
(# ENIs × (IPs/ENI − 1)) + 2, published as a convenience table ineni-max-pods.txtin theamazon-vpc-cni-k8srepo. This is the floor the AMI falls back to if nothing above was set.
The consequence for dense multi-tenant clusters is specific and easy to miss. If you run a stock Managed Node Group on 16xl-and-up Graviton, the
eni-max-pods.txt number says 737 — but the Managed Node Group cap says 250, and 250 is what kubelet actually starts with. Pods past that number are simply not schedulable on the node, even though the CNI could have supported far more and the instance family was picked precisely for its network capacity. The capacity looks present in the table but is invisible to the scheduler.To actually reach the density the triangle and network capacity permit, the path on Managed Node Groups is a custom AMI in the launch template, passing
--use-max-pods false and --kubelet-extra-args '--max-pods=…' to bootstrap.sh. Self-managed nodes skip this dance — nodeadm maxPodsExpression or a direct --max-pods flag is enough.This is also why the earlier statement that "the instance choice and the density target have to be picked together" is more than a soft guideline: choosing an instance size and choosing the
maxPods path to reach its capacity are the same decision, and getting either wrong means capacity you thought you had is simply not available to the scheduler.One last thing to keep in mind once
maxPods is set correctly — and this is the request-driven ceiling foreshadowed in the triangle discussion above: the actual pod density on a node is min(maxPods, allocatable / per-pod request), where the second term is computed separately for CPU and memory and bound by the tighter of the two. A concrete feel for this: if the agent pod requests 500m CPU and the node has 64 allocatable vCPUs, the scheduler places at most 128 pods/node, regardless of what the kubelet says it could hold. If per-pod resource requests are conservative, maxPods is what binds. If they are liberal, the request-driven ceiling is what binds — and maxPods ends up being headroom you paid for but never used. The density target you commit to has to clear both.Prefix Mode and Its Quieter Failure Mode
By default, VPC CNI assigns pod IPs as secondary IPs on ENIs, one at a time. On a Graviton node, that caps pods-per-node well below 200 — the ENI-times-secondary-IP arithmetic simply doesn't get you there. To reach the density the triangle allows (and that
maxPods permits), you turn on VPC CNI prefix mode , which switches the ENI from "one IP at a time" to "one /28 prefix at a time" — 16 IPs in a single allocation. That multiplies per-ENI capacity by roughly an order of magnitude and unlocks the 200+ pods-per-node regime.This is the piece the networking section gestured at when it flagged CIDR Reservation as a decision that had to be made at subnet creation time. Here is why.
The fragmentation
A
/28 prefix has to be contiguous and 16-aligned inside the subnet. You cannot hand VPC CNI a /28-sized hole that starts on a non-/28 boundary; the alignment is a hardware-level constraint, not a CNI preference.In a long-running subnet, secondary IP allocation from pod lifecycle churn slowly scatters addresses across the subnet. A pod starts, takes an IP; another pod ends, releases a different IP; a new ENI comes up, grabs a few IPs from a fresh block. Over days and weeks, the subnet accumulates gaps — plenty of free IPs in total, but very few of them sitting in aligned 16-address runs.
This is the "quieter failure mode" the section title refers to. It is invisible under steady-state load. The subnet looks fine. Free IP count looks fine. Then scaling kicks in — a deployment ramps, Karpenter asks for new nodes, VPC CNI tries to attach a new ENI and allocate a fresh
/28 for it — and the allocation fails with:1
2
InsufficientCidrBlocks: There are not enough free cidr blocks
in the specified subnet to satisfy the requestThe symptom is specific and easy to miss: existing pods are unaffected. New pods cannot come up. Scaling itself is blocked. By the time it surfaces, the subnet has already been poisoned by churn, and no operational action brings the fragmentation back.
The fix has to be pre-committed
The mechanism that prevents this is VPC Subnet CIDR Reservation : carve out a range inside the subnet that is reserved exclusively for prefix allocation. Secondary IPs can never consume addresses inside a reservation, so the reserved range stays clean and
/28-aligned forever.1
2
aws ec2 create-subnet-cidr-reservation \
--subnet-id <id> --cidr <block> --reservation-type prefixThe non-obvious part is the timing. Reservations on an already-running subnet do not repair existing fragmentation — they only protect whatever is reserved from that point forward. By the time
InsufficientCidrBlocks shows up in the logs, the subnet is already in a state no in-place operation can reverse.Which is why the networking section treated it as a networking decision rather than a pod-density one. The reservation has to exist at
CreateSubnet, before any pod has ever been scheduled onto that subnet. How big a reservation? That depends on the density target, expected pod churn, and how much headroom you want against future growth — all of which are pod-density-layer inputs, decided at networking-layer time.What Pod Density Taught Us
First, maximum density and optimal density are different numbers. The instance-level ceiling on pods-per-node tells you what is possible; the intersection of instance family, network capacity, and cost-per-pod tells you where to actually sit. Pushing for the ceiling sacrifices scheduling flexibility and per-pod resource budget in ways that don't show up until workloads run on it.
Second, the most dangerous failure modes at scale are the ones that accumulate quietly and only surface during scaling. Subnet fragmentation is the archetype: steady-state traffic never reveals it, and by the time a scale event triggers
InsufficientCidrBlocks, no runtime fix exists. The only defense is a decision made far upstream, at subnet creation, when the problem is abstract and it is tempting to skip. Scale-driven architecture means recognizing those upstream decisions while the problem still looks abstract — and committing to them before the cost of wrong shows up as InsufficientCidrBlocks in a scale event.Part 1 was about limits you can measure — a /16, 250 pods, a
/28 alignment. Part 2 is about limits you can't. Once IP space and density are sorted, the constraints stop being numeric and start being behavioral. Trust boundaries don't have a quota; they have a blast radius, and it is decided by the isolation boundary your kernel offers. Persistent state doesn't have an address-space cap; it has a consistency model, and it is decided by the filesystem you pick. Those choices look softer than the ones in Part 1 — and they're the ones most likely to age badly.Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article