GPUs on AWS, above the cluster
The previous page starts at an EC2 instance that already exists and stops at a CUDA kernel: the GPU Operator, the sharing modes, the silent NCCL fallback, the metric that reports a starved GPU as busy. It is the layer between the instance and the kernel, and it assumes the instance.
That assumption is where most GPU programmes actually fail. Three questions sit outside it:
- Can you have the accelerator at all, on the day you need it?
- Who gets it when two teams ask for the same eight?
- What did the hour buy, given that an idle GPU and a stalled GPU cost exactly the same?
None of the three is a kernel problem, and none of them is answered by a chart of GPU utilisation. This page is the layer above and around the cluster.
Read the right-hand column first. Every gap in it is work someone on your team is already doing, usually without having decided to.
1. Capacity: the constraint that is not technical
There are five ways to pay for an accelerator on AWS and one way to avoid holding one at all. They differ in a single dimension that matters more than price: how it can be taken away from you.
| path | what you are buying | how it ends | fits |
|---|---|---|---|
| On-Demand | an instance, if one exists | you stop it | experiments, small inference |
| Capacity Block for ML | a named block of accelerators between two timestamps, paid up front | the end timestamp, finished or not | a training run with a date |
| On-Demand Capacity Reservation | held capacity in one AZ for one instance type | you release it | steady serving |
| Savings Plan / Reserved | a discount on spend, not capacity | the term | anything long-lived |
| Spot | someone else's spare capacity | two minutes' notice | restartable work that checkpoints |
| Bedrock / managed endpoint | tokens, not hours | you stop calling | when GPU operations is not your job |
The rows are combined in practice — a Savings Plan is a discount over an ODCR's usage, not an alternative to it — but the failure modes do not combine. A workload has to be able to survive the way its capacity ends, and that is a property of the job, not of the purchase.
The quota that says no before capacity does
Before a single instance is refused for capacity reasons, it is refused for quota reasons, and the quota is not denominated in the unit you are thinking in. The EC2 accelerated quotas —
Running On-Demand P instances, All P Spot Instance Requests and their G-family equivalents — are counted in vCPUs, not in instances and certainly not in GPUs.A
p5.48xlarge is 192 vCPUs and 8 H100s. Eight of them — one modest training run — is 1,536 vCPUs, and a fresh account's limit for that family is frequently zero. The error does not mention GPUs, the increase is reviewed by a human, and the review is not instant.1
2
3
4
5
6
7
8
9
# Ask for the number before the project plan depends on it.
aws service-quotas get-service-quota \
--service-code ec2 \
--quota-code L-417A185B # Running On-Demand P instances (vCPUs)
# The Spot family has its own quota, and it is a separate request.
aws service-quotas get-service-quota \
--service-code ec2 \
--quota-code L-7212CCBC # All P Spot Instance Requests (vCPUs)
Quota codes move; look yours up with
list-service-quotas rather than trusting the two above. The point stands whatever the codes are: the first capacity conversation is with Service Quotas, and it is a lead-time item.What a Capacity Block actually is
The shape is visible in the API, which is worth reading before the pricing page because it tells you what you may ask for:
1
2
3
4
5
6
7
# "When can I have 16 p5.48xlarge together, for a week, in the next month?"
aws ec2 describe-capacity-block-offerings \
--instance-type p5.48xlarge \
--instance-count 16 \
--capacity-duration-hours 168 \
--start-date-range 2026-10-01T00:00:00Z \
--end-date-range 2026-11-01T00:00:00Z
DescribeCapacityBlockOfferings takes a duration in 1-day increments up to 14 days, and 7-day increments up to 182; a block holds up to 64 instances, and an account up to 256 across blocks. Each offering that comes back carries a start and end date, an Availability Zone, an upfront fee, and — for UltraServer offerings — an UltraServer type and count. You purchase an offering by its id; the capacity arrives at the start date and leaves at the end date.Three consequences that are easy to miss until the first block:
- The block is a deadline, not a lease. Work that has not checkpointed by the end timestamp is work you paid for and threw away.
- It lands in one Availability Zone, which is the same AZ your data had better be in. A cross-AZ training set is a bandwidth bill and a latency tax.
- The upfront fee is charged when you purchase, not when you use it, so an unused block is the most expensive idle capacity in the account.
The API that does not exist
DescribeInstanceTypeOfferings tells you a type is offered in an Availability Zone. It does not tell you whether there is any right now. There is no "is there capacity" API: InsufficientInstanceCapacity — returned from an attempt to launch — is the availability signal, which means the only way to find out is to try, and the only way to hold the answer is to reserve.This is the first wish on the GPU wish list beside this page, and it is the one that changes plans rather than dashboards.
2. Placement: where they land is a performance decision
Eight instances in a region is not the same purchase as eight instances in one cluster placement group. The collective only runs at wire speed if the instances are near each other, and "near" is a property of the physical network, not of the VPC.
AWS does expose the topology:
1
2
3
aws ec2 describe-instance-topology \
--filters Name=instance-type,Values=p5.48xlarge \
--query 'Instances[].{id:InstanceId,az:AvailabilityZone,group:GroupName,nodes:NetworkNodes}'
InstanceTopology returns an AvailabilityZone, the placement GroupName, a CapacityBlockId for UltraServer instances, and NetworkNodes — an ordered list of hashed strings, one per network tier, salted per account. Two instances that share the deepest string share the deepest switch.That is genuinely useful and it stops one layer short of being usable: nothing turns it into a node label. Both schedulers that can act on topology want a label —
- Kueue's
Topologyobject lists node labels, coarsest tier first; - Volcano's
networkTopologyconstrains a job to a tier of its HyperNode tree —
and populating that label is a controller you write, from an API that returns hashes. Until you write it, topology-aware scheduling on EKS degrades quietly to "somewhere in the same Availability Zone", which is exactly the failure the NCCL gate on the previous page is designed to catch after the fact.
1
2
3
4
5
6
7
8
9
10
11
12
13
# The crude version of the controller nobody ships: label each node with the
# deepest network tier its instance sits under, so a scheduler can keep a gang
# together. Run it after a node group scales; it is a stop-gap, not a design.
aws ec2 describe-instance-topology --output json \
| jq -r '.Instances[] | "\(.InstanceId) \(.NetworkNodes[-1])"' > /tmp/topology
kubectl get nodes -o json \
| jq -r '.items[] | "\(.spec.providerID | split("/") | last) \(.metadata.name)"' \
| while read -r id name; do
tier=$(awk -v i="$id" '$1 == i {print $2}' /tmp/topology)
[ -n "$tier" ] && kubectl label node "$name" \
"aws-demo.io/network-node-layer-2=$tier" --overwrite
done
One layer further down, the same idea repeats inside the box: a GPU on one NUMA socket talking to an EFA device attached to the other crosses the CPU interconnect on every packet. The kubelet knobs for that — the CPU Manager's
static policy and the Topology Manager's single-numa-node policy — are node bootstrap configuration, so they belong to the launch template, not to the pod spec, and they are the difference between a 30% slower run and a run whose profile looks inexplicable.3. Sharing the pool: what Kubernetes does not give you
Two teams, one pool. Kubernetes' own tools are the wrong shape for this and it is worth being precise about why:
| you want | Kubernetes offers | why it does not work |
|---|---|---|
| a floor per team | ResourceQuota | a ceiling nobody can lend; idle quota is wasted, not borrowed |
| "research may use idle inference capacity" | nothing | there is no cohort, no borrowing, no lending limit |
| all four workers or none | parallelism: 4 | starts three, holds three GPUs, waits for the fourth |
| evict a job, not a pod | PriorityClass | preempts one pod; the survivors keep their GPUs and make no progress |
| half a GPU | nvidia.com/gpu: 1 | integers only, no fractions, no overcommit |
Every row is solved by something you install and operate. On EKS none of it is an add-on: Kueue, Volcano and HAMi are all "run it yourself".
Quota that can be lent, and taken back
The three manifests in this section live under
k8s/gpu/not-aws/ , and the folder name is the argument: everything else in this repository is an AWS-managed service, and these are the counted exceptions. Two of them (the DCGM scrape and the productive-hours rules) have a partial AWS answer — CloudWatch for "is the GPU busy", Amazon Managed Service for Prometheus when the question is "is it productive". Two of them do not: nothing in EKS expresses a lending limit, and nothing in batch/v1 expresses all-or-nothing placement. The AWS-managed alternative for those two is to move the queue out of the cluster entirely — AWS Batch job queues with fair-share scheduling, or SageMaker HyperPod — which is a different product rather than a drop-in, and that is the honest version of the claim.k8s/gpu/not-aws/08-kueue-gpu-quota.yaml is the whole model in one file: three ResourceFlavors — because an accelerator bought as a Capacity Block, as reserved On-Demand, and as Spot are three different things to schedule onto — and two ClusterQueues in one cohort.1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
name: inference
spec:
cohortName: accelerators # the pool the two queues borrow across
preemption:
reclaimWithinCohort: LowerPriority # take back what was lent
withinClusterQueue: LowerPriority
resourceGroups:
- coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
flavors:
- name: p5-on-demand
resources:
- name: nvidia.com/gpu
nominalQuota: 32 # 4 nodes x 8
lendingLimit: 16 # at most half is lent to research
lendingLimit is the line that makes the arrangement survive contact with the inference team: they keep a guaranteed 16 no matter how much research has borrowed, and the other 16 come back by preemption rather than by a meeting. The research queue sets flavorFungibility.whenCanBorrow: TryNextFlavor so a job that cannot have a p5 today takes Spot g6 capacity instead of waiting for one — the queue prefers a worse GPU now to a better GPU later, which is the right default for research and the wrong one for serving.Priority belongs to the workload, not the pod:
1
2
3
4
5
6
apiVersion: kueue.x-k8s.io/v1beta2
kind: WorkloadPriorityClass
metadata:
name: research-batch
value: 100
description: "Preemptible. Must checkpoint; will be evicted whole."
The gang, and the loss that has no metric
k8s/gpu/not-aws/09-gang-training-job.yaml is four workers that are worth nothing until all four run:1
2
3
4
5
6
7
8
9
10
11
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
spec:
schedulerName: volcano
minAvailable: 4 # admit four or admit none
networkTopology:
mode: hard # refuse to spread the gang beyond one tier
highestTierAllowed: 2
policies:
- event: PodEvicted
action: RestartJob # a lost member invalidates the run, not the pod
Without
minAvailable, three workers start and sit in NCCL rendezvous holding three GPUs while the fourth waits for a node. Nothing errors. The three that started report high CPU and zero tensor activity, so a utilisation dashboard shows a busy cluster, and the invoice shows three GPUs at full price producing nothing. This is the second loss in the funnel below, and it is the one with no first-party signal anywhere in AWS.policies matters as much as minAvailable: when a member is evicted — Spot reclaim, node repair, preemption — the job is restarted, because in a synchronous data-parallel run the survivors cannot make progress alone.Fractions, when a whole GPU is too much
Time slicing multiplies the advertised count without partitioning memory; MIG partitions in hardware with fixed profiles; both are covered on the previous page . The third option is HAMi (v2.9.0 at the time of writing), which adds device memory and compute limits as ordinary resources:
1
2
3
4
5
resources:
limits:
nvidia.com/gpu: 1 # one physical GPU, shared
nvidia.com/gpumem: 3000 # 3 GiB of it, enforced
nvidia.com/gpucores: 30 # 30% of the SMs
That is precisely the defect that makes plain time slicing unsafe between tenants, fixed by a scheduler and device plugin you run yourself. AWS manages none of it; the trade is isolation for one more component you own.
Dynamic Resource Allocation is where this is heading in upstream Kubernetes — devices requested as claims against a
DeviceClass rather than as opaque integer resources, with the vendor driver deciding what "half" means. It was not exercised for this page: the lab cluster runs 1.33 with the resource.k8s.io APIs disabled, so everything above is device-plugin-era mechanics and says so.4. Keeping it alive
At fleet scale the interesting failure is not that a GPU is slow, it is that a GPU is broken and the job does not know. An
Xid in the kernel log, an ECC error retiring a page, a link that dropped from NVLink to PCIe: each degrades or kills one worker, and a synchronous training job degrades or dies with it.AWS's answer on EKS is the node monitoring agent and node auto repair: the agent surfaces hardware faults as node conditions, and auto repair replaces a node that stays unhealthy. That is the right primitive and it stops at the node boundary. What it does not do is tell the workload anything: the job finds out by losing a rank. Getting from "node replaced" to "run resumed from the last checkpoint" is your controller, unless you adopt SageMaker HyperPod, which ships deep health checks and auto-resume and asks you to accept a second control plane beside the one your platform already runs.
Which leaves checkpointing as the load-bearing decision of the whole design:
| capacity path | the interval that matters |
|---|---|
| Spot | shorter than the notice you are given, with the write finished before the instance goes |
| Capacity Block | such that a checkpoint lands before the block's end timestamp |
| On-Demand / reserved | such that a failed node costs one interval, not one run |
The rule that follows is blunt: a job that cannot resume from a checkpoint can only run on capacity that never goes away, which is the most expensive capacity there is. Checkpoint cadence is a purchasing decision disguised as an engineering one. Where the checkpoint goes — and why it must not block the step — is the storage page .
5. The hours you actually paid for
Four numbers, three gaps, and each gap has a different owner:
| gap | what it is | whose problem |
|---|---|---|
| present → allocated | capacity bought that nobody asked for | finance and capacity planning |
| allocated → busy | image pull, dataset hydration, a half gang, a node being replaced | the platform |
| busy → productive | memory-bound stalls, NCCL on TCP, the dataloader, a rolled-back step | the workload |
k8s/gpu/not-aws/10-productive-hours-rules.yaml records all four from metrics the GPU Operator already exports:1
2
3
4
5
6
- record: gpu:allocated:count
expr: count(DCGM_FI_DEV_GPU_UTIL{exported_pod!=""})
- record: gpu:busy:count
expr: count(DCGM_FI_DEV_GPU_UTIL{exported_pod!=""} > 10)
- record: gpu:productive:count
expr: count(DCGM_FI_PROF_PIPE_TENSOR_ACTIVE{exported_pod!=""} > 0.15)
and the alert worth having before any of the dashboards:
1
2
3
4
5
- alert: GpuIdleWhileWorkloadsPending
expr: |
(gpu:present:count - gpu:allocated:count) > 0
and on() (sum(kueue_pending_workloads) > 0)
for: 20m
On the AWS target these rules are not a
PrometheusRule in the cluster: the same groups: block loads into the Amazon Managed Service for Prometheus ruler as a rule groups namespace, the AMP managed collector scrapes the DCGM exporter and Kueue's metrics, and the dashboards are Amazon Managed Grafana's. The PromQL does not change.Idle GPUs and a queue is never a capacity problem. It is a flavor that matches no node, a quota, a taint, or a gang that cannot be completed — and it is the single most common reason a team asks for more hardware than it needs.
Two traps in the accounting itself:
nvidia-smi's memory figure overstates the workload. CUDA reserves a context pool on first use, so a process that needs 512 MB can report gigabytes. Sizing a MIG profile or a HAMigpumemlimit from that number buys hardware to hold an allocator's reservation.- The device-to-pod mapping is not Kubernetes' own. Kubernetes does not track which GPU UUID a pod received; the exporter's pod labels are how you join a device to a tenant, and if the exporter's Kubernetes mapping is off, every per-team number above silently becomes a per-node number. The label is
podas the exporter emits it andexported_podafter the scraper (the lab's Prometheus, or the AMP managed collector) resolves the collision with its own — check one series before trusting a dashboard built on either.
The bill does not split the way the cluster does
Cost Explorer bills you for an instance. Split cost allocation data for EKS splits an instance's cost across pods by CPU and memory — not by GPU — so on a node whose entire value is eight accelerators, the split is performed on the two resources nobody is contending for. Per-team GPU cost is therefore a reconstruction: allocated GPU-hours per namespace from the metrics above, multiplied by a price you keep in a config file, reconciled against the invoice at the end of the month.
That reconstruction is worth doing anyway, because the number it produces — cost per productive hour, or for inference cost per million tokens — is the only one that compares two clusters, two instance types or two model servers. Utilisation compares nothing: a fleet at 85% utilisation and 20% productive does not need more GPUs.
No prices appear on this page. They move, they differ by region and commitment, and a wrong one printed confidently is worse than none.
Triage, in order
When someone says the GPUs are slow and asks for more of them:
- Is anything pending while GPUs sit unallocated? If yes, stop: it is a scheduling fault — flavor, quota, taint, or an incomplete gang. No amount of capacity fixes it.
- Are allocated GPUs busy? If not: image pull, dataset hydration, a gang waiting for its last member, a node being replaced.
- Are busy GPUs productive? Tensor-pipe activity against utilisation, per the previous page . Busy and unproductive is a data path or a collective, not a kernel.
- Is the collective on EFA? Read the transport line in the NCCL log. A silent fallback to TCP costs more than any kernel-level inefficiency you will find afterwards.
- How fast can you feed it? Benchmark the dataloader with no model attached before asking why the model is slow.
- Only now, buy capacity or profile a kernel.
Steps 1 and 2 belong to this page, 3 to 5 to the previous one, and step 6 is where most teams start.
Verified, and not
Validated. The three new manifests —
not-aws/08-kueue-gpu-quota.yaml , not-aws/09-gang-training-job.yaml and not-aws/10-productive-hours-rules.yaml — pass kubectl apply --dry-run=server against a lab cluster (kind, Kubernetes 1.33) carrying Kueue v1beta2, Volcano and the kube-prometheus-stack admission webhook. Server-side validation is the real check: a deliberate typo in a ResourceFlavor field and a deliberate syntax error in one of the PromQL expressions were both rejected, so the nine recording rules and alerts above are at least parseable PromQL, and the Kueue and Volcano objects match the CRDs' schemas. The LocalQueue was validated against an existing namespace, since ml does not exist on that cluster.Checked against the SDK. Every EC2 API claim on this page was read from
aws-sdk-go-v2/service/ec2 v1.232.0 in the local module cache rather than from memory: DescribeCapacityBlockOfferings' duration increments (1-day to 14 days, 7-day to 182) and counts (64 per block, 256 per account); CapacityBlockOffering's start and end dates, Availability Zone, upfront fee and UltraServer fields; InstanceTopology's hashed NetworkNodes, GroupName and CapacityBlockId; and GpuInfo, which carries a count, a manufacturer, a name and a memory size — and nothing about bandwidth, interconnect or MIG.Not verified. No accelerated instance was launched, no Capacity Block was purchased, and no Spot GPU node was reclaimed, for this page or any other in this repository: its Terraform provisions CPU node groups and has never been applied. The EKS node monitoring agent's condition names, node auto repair's behaviour, SageMaker HyperPod's health checks, Container Insights' GPU metric names and the Service Quotas codes are from documentation, not from a run, and all of them move. Prices are deliberately absent.
Grounding. The host, collective and storage material is indexed from AI Systems Performance Engineering (Chris Fregly, O'Reilly), chapters 3 , 4 and 5 . The multi-tenancy failure modes — the CUDA context pool inflating reported memory, the missing device-to-pod mapping, what each isolation mode does not isolate — come from Jimmy Song's GPU multi-tenancy platforms book. The busy-versus-productive argument is Arjun Kaarat's, by way of Song's essay on it ; the NUMA and PCIe placement argument is Manan Paliwal's . Component names and resource names were read from local checkouts of HAMi v2.9.0 and the NVIDIA GPU Operator , and from the Kueue and Volcano CRDs on the lab cluster. The mapping onto AWS, the five-layer frame, the funnel and the triage order are this page's own.
AWS-only revision (2026-09-25). Where the recording rules and alerts run moved to AWS-managed services: the AMP managed collector and ruler, with Amazon Managed Grafana for the dashboards. The rule expressions are unchanged, and the validation above is still the lab's
kubectl apply --dry-run=server; the rules were not loaded into an AMP workspace, and that the collector keeps the exported_pod relabelling behaviour is documentation-sourced, not checked. Nothing was deployed, and AWS documentation in this area changes monthly.Questions this page should answer
If the page did its work, each of these is answerable from it alone. The section named after each question is where the argument lives, not a hint.
- Five ways to hold an accelerator differ in one dimension that matters more than price. Which, and what property of the job has to match it? — Capacity
- What unit are the accelerated EC2 quotas counted in, how many of that unit is one modest eight-instance training run, and why does the refusal never mention a GPU? — The quota that says no before capacity does
- Name the three things about a Capacity Block that are easy to miss until you have bought one. — What a Capacity Block actually is
- AWS returns instance topology and no scheduler can use it. What is missing in between, and what does topology-aware scheduling quietly degrade to? — Placement
- Three of four gang members start. What does the utilisation dashboard show, what does the invoice show, and which two fields in the job prevent it? — The gang
- Why is checkpoint cadence a purchasing decision rather than an engineering one? — Keeping it alive
- Split cost allocation divides a node's cost by CPU and memory. Why is that the wrong split on a GPU node, and which number is the only one that compares two clusters? — The bill does not split the way the cluster does
Next
The gaps in the right-hand column of the first figure are not rhetorical: each one is a thing AWS could close. They are written up as twelve wishes , one AWS service each, and as twelve polls — one per wish — that test whether the gap is real before anyone files it, and as twelve comments for raising the same gaps in threads someone else started.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article