The Hardest Problem for ISVs at Hyperscale Isn’t Technology - It’s Capacity
Patterns my team keeps seeing across large ISVs running at scale on AWS.
Series: Operating at Hyperscale (7 articles)
- …
- 5The Hardest Problem for ISVs at Hyperscale Isn’t Technology - It’s Capacity This article
My team supports a portfolio of large independent software vendors (ISVs) on AWS: SaaS platforms serving millions of users, AI companies training and serving their own models, developer platforms deploying functions by the millions. Different products, wildly different architectures. But when we look across all of them, the same class of problem shows up again and again, and it isn't a technology problem. These are excellent engineering teams. The recurring wall is capacity: getting the right compute, in the right place, in the right shape, at the right time.
At normal scale, capacity is invisible. You auto-scale, you buy a Savings Plan, and you never wonder whether the instances exist. At hyperscale, that assumption breaks in four distinct ways. Here are the four patterns we see most, and the practices that help teams work through them.
A note before we start: none of the approaches below guarantee capacity. The availability of GPUs, instances, and quota is always subject to regional supply and standard review. What these practices do is improve the quality of your request, reduce how much you depend on any single pool, and keep you resilient when the answer is "not right now." The teams that do best are the ones who treat capacity as something to plan for and manage, not something they can assume.
Pattern 1: GPU Scarcity Is Managed Through Lead Time and Forecasting Discipline, Not Purchase Orders
The single most contended resource in the cloud right now is the newest-generation GPU. When an AI-focused ISV needs to stand up or grow a training or inference cluster, "just launch more" is not a plan. The units are large (hundreds of instances), scarce, and demand is lumpy and deadline-driven, often tied to a model launch or a contract expiration.
What consistently helps is treating capacity as a planning problem rather than a one-off request:
- Translate an open-ended ask into a concrete ramp. A vague "we need 800 to 1,700 GPUs sometime next quarter" is hard to act on. Broken into "N instances per month starting in month X, with regional flexibility," it becomes something that can be reviewed and planned against. In our experience, a well-structured, regionally flexible ask moves through review far more smoothly than an open-ended one, though timelines always depend on supply and region.
- Start a full cycle ahead of the deadline. The teams best positioned for a next-generation cluster are the ones planning against the next contract expiration, not scrambling at the current one. Early planning does not guarantee availability, but it materially improves your options. One practical constraint to know: EC2 Capacity Blocks for ML can be booked with a start time up to eight weeks in advance (current limits here ), so for anything beyond that horizon, your lead time lives in forecasting and communicating demand early, not in the reservation itself.
- Fix the forecast before you submit it. A recurring failure mode: teams carry aggressive "just in case" headroom that projects exponential growth against what is really linear usage. Left uncorrected, that forecast is hard to fulfill and ties up scarce supply. Correcting the methodology, net-new capacity with sane headroom rather than double-counted buffers, is often the highest-leverage thing you can do.
- Release what you won't use. When a timeline slips, giving reserved capacity back rather than holding it keeps you a trusted planning partner for the next window instead of burning goodwill and budget.
AWS tooling that fits: EC2 Capacity Blocks for ML let you reserve a GPU cluster for a defined future window (see the current durations and limits ) with a known start date, and the instances are colocated in EC2 UltraClusters for low-latency networking. That scheduled, future-dated model fits deadline-driven training far better than hoping On-Demand capacity exists at launch time. On-Demand Capacity Reservations hold capacity in a specific Availability Zone with flexible duration and no fixed end date, and SageMaker HyperPod provides resilient managed cluster infrastructure with automated health checks and node replacement (job auto-resume works when checkpointing is enabled), which lowers the buffer you need to hold. A recurring quarterly capacity-planning cadence, with forecasts shared ahead of time, turns this from reactive constraint-response into forward planning.
Pattern 2: At Scale, Reservations Behave Differently Than the Docs Suggest
Here is the pattern that surprises teams most: reserving capacity and using capacity are not the same thing at hyperscale. I have repeatedly seen reservations that look healthy but do not deliver:
- An On-Demand Capacity Reservation showing "active" with available capacity, where every launch failed with
InsufficientInstanceCapacitybecause the underlying pool had drained. The fix was to cancel that reservation, relaunch in a different Availability Zone, and stop paying for the unusable one. (This applies to ODCRs; Capacity Blocks cannot be cancelled.) - A held On-Demand Capacity Reservation where only a fraction of the committed instances could launch because failed hardware had quietly reduced the pool below the committed level. (Capacity Blocks handle impairment differently: for an UltraServer Capacity Block, AWS notifies you but generally does not terminate your instances, and you can request remediation by terminating all instances on the block and opening an AWS Support case. See the Capacity Blocks considerations .)
- Teams wanting to move reserved capacity between production, staging, and research accounts, with no simple way to do it, resorting to risky return-and-redeliver.
These examples are worth sitting with, because they show that reserved capacity is not a guarantee of launchable capacity. At scale you have to manage reservations as a live operational surface, not a set-and-forget purchase. What helps:
- Design for fungibility (for On-Demand Capacity Reservations). Use a split, share, and group pattern : split a large ODCR, share the pieces across accounts, and regroup them in the consuming account, so capacity can move between environments without risky returns. A few constraints worth knowing before you build this topology: ODCR sharing works only for shared-tenancy instances, is not available to brand-new accounts or those with limited billing history, and you can share reservations in the
activeorscheduledstate. Note that EC2 Capacity Blocks behave differently: they cannot be split or moved, and standard Capacity Blocks can only be shared within the same AWS Organization (UltraServer Capacity Blocks cannot be shared at all). So the split-share-group pattern is an ODCR technique, not a Capacity Blocks one. - Match reservations to real instance shapes, and keep some flexibility on instance type and region to improve the odds of fulfillment in constrained pools.
- Treat a reservation failure as an operational incident with the same urgency as a production one, because at these revenue levels, it is one.
AWS tooling that fits: Capacity Reservation Groups and AWS Resource Access Manager for cross-account sharing of ODCRs, On-Demand Capacity Reservations for AZ-pinned capacity with
open or targeted instance-match-criteria (use targeted to prevent unintended use of a reservation), and the documented ODCR split/move/modify operations for reshaping capacity. The meta-point: read the sharing and lifecycle constraints for the specific reservation type you are using, ODCRs and Capacity Blocks differ, before you build the reservation topology, not after a launch fails.Pattern 3: "Capacity" Often Means Quota, Not Hardware
A large share of what gets reported as a "capacity problem" is actually an account limit or service quota problem, and it is the one teams most often try to brute-force. The tells we look for:
- A platform ISV filing thousands of storage-limit-increase tickets over a year, with one account growing more than a terabyte in a single day, because the architecture leaned on a per-account quota that was never meant to scale that way.
- Serverless control-plane rate limits (for example, needing on the order of 100 TPS against a much lower default), network-interface throttling, and non-standard scaling quotas that require manual service-team review.
- Inference token/throughput ceilings (for Amazon Bedrock, expressed as tokens per minute, or TPM) hit during model launches, where the "capacity" being exhausted is a quota profile, not a GPU pool.
The wrong move is an endless stream of limit-increase requests. The better move is to ask whether the architecture is fighting a quota that was never designed for that pattern, and fix it structurally:
- When you are filing the same increase repeatedly, that is a design signal. In the storage example, the durable fix was an architectural change (moving to self-managed storage) that removed the per-account ceiling entirely, turning a daily operational fire into a non-issue.
- Engage the service team early on non-standard quotas rather than discovering the manual-review requirement at the deadline.
- Distinguish quota ceilings from real capacity during incidents. We have seen inference errors blamed on capacity that were actually traffic hitting the wrong quota profile while an allocated one sat idle.
AWS tooling that fits: Service Quotas for visibility and increases, and more importantly a habit of pattern-spotting: when quota requests recur, escalate to an architecture conversation. The best outcome is not a bigger limit; it is an architecture that no longer depends on that limit.
Pattern 4: Before You Ask for More, Right-Shape What You Have
The quietest capacity problem is the most common: teams are not short on capacity, they are carrying it in the wrong shape. Old-generation instances where newer ones are cheaper and faster. Reservations that no longer match the workload. And, over and over, storage nobody is watching:
- Hundreds of detached volumes (hundreds of TiB) left orphaned because a Kubernetes reclaim policy retained them after the pods were gone: a low-six-figure annual bill for storage serving nothing.
- A storage-to-compute ratio analysis is a fast way to find these. Accounts with disproportionate storage spend relative to compute are orphan-candidate hotspots worth scanning first.
- Large fleets still on prior-generation instances, where a migration to current-generation (including Graviton) delivers price-performance gains at scale, sometimes across hundreds of thousands of instances and many regions.
The discipline: make waste visible, quantify it in dollars, and give the team self-service verification so the cleanup sticks after you leave the room. A lot of "we need more capacity" is really "we are using what we have inefficiently."
AWS tooling that fits: AWS Compute Optimizer for right-sizing, Cost Optimization Hub for consolidated savings opportunities, Trusted Advisor for idle and underutilized resource checks, Savings Plans and Reserved Instances for commitment coverage, and AWS Graviton for price-performance on compatible workloads. For the storage-orphan case specifically, enabling Cost Optimization Hub at the payer gives you ongoing detection instead of one-off scans.
The Pattern Behind the Patterns
Four different-looking problems, GPU scarcity, reservation mechanics, quota ceilings, and wrong-shaped capacity, but the same underlying discipline runs through all of them. You cannot control supply. You can control how you forecast, request, reserve, and right-shape, and that discipline is what determines how well you weather constraint.
- Plan a cycle ahead. Scarcity at hyperscale is a lead-time problem. Whoever plans earliest has the most options.
- Manage reservations as a live operational surface, not a purchase you make once.
- Treat recurring quota requests as an architecture signal, not a ticket to refile.
- Right-shape before you scale up. Reclaim and modernize what you have before you ask for more.
None of this is exotic. It is operational discipline applied to a resource that most teams are trained to treat as infinite and instant. At hyperscale, compute is neither, and the teams that internalize that early spend far less time firefighting capacity at the worst possible moment.
If you are running large-scale or AI-heavy workloads on AWS and wrestling with any of these, GPU lead times, reservation fungibility, quota ceilings, or reclaiming what you already pay for, I would like to hear what is working for you.
References (AWS documentation)
Where this article states specific AWS behavior or limits, see the official documentation for the current details:
- EC2 Capacity Blocks for ML and how Capacity Blocks work / current limits
- Sharing Capacity Blocks
- On-Demand Capacity Reservations and shared Capacity Reservations / prerequisites
- Manage ODCRs with split, move, and modify
- Troubleshooting
InsufficientInstanceCapacity - Amazon SageMaker HyperPod
- AWS Service Quotas
- AWS Compute Optimizer
- Cost Optimization Hub
- AWS Trusted Advisor
- Savings Plans and AWS Graviton
About the author: The author is an Enterprise Support Manager at AWS, supporting many of the leading independent software vendors that build their platforms on AWS. This vantage point across a portfolio of multi-tenant SaaS providers informs the generalized patterns described here; nothing in this post reflects any individual customer's confidential architecture or operations.
Series: Operating at Hyperscale (7 articles)
- …
- 5The Hardest Problem for ISVs at Hyperscale Isn’t Technology - It’s Capacity This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article