Distributed LLM Inference on AWS: Serving a 70B Model Across Multi-GPU and Multi-Node with vLLM on EKS

A practical walkthrough of serving a 70B open-weights model on AWS: how to work out your GPU memory budget, how to decide what to split across which GPUs, how to wire up the fast networking on Amazon EKS, and how to route, autoscale, benchmark and cost-model the result.

Mohammad AlYasfo

Senior Software Engineer | AI & FinTech | .NET, Python | Cloud

Published Sep 16, 2026

Last modified Sep 16, 2026

Most teams run into distributed inference the same way. A model that ran fine on one GPU during a proof of concept now has to serve real traffic, and the first honest capacity calculation says it won't fit. Or it fits, but there's no memory left for the per-conversation cache, and the number of users you can serve at once drops to a handful.

This is the approach I use when a team hits that point on AWS.

In short, work out the memory budget first, because everything else follows from it. Split one copy of the model across the GPUs inside a single machine, and scale out by adding more whole copies on more machines. Only spread a single model across machines when it really won't fit in one. After that, keep an eye on three numbers: queue depth, how long users wait, and what a million tokens costs.

You'll get the most out of this if you're comfortable with Kubernetes (Deployments, resource limits, readiness probes) and have run a model server before. You don't need prior experience with NCCL, EFA or tensor parallelism. I explain those as they come up.

The running example is Llama 3.1 70B in FP16 on a p5.48xlarge (eight NVIDIA H100 GPUs with 80 GB each), served by vLLM on Amazon EKS. The same reasoning applies to other large open-weights models, to the G6e, P4d/P4de and P5e/P5en families, and to AWS Trainium and AWS Inferentia.

A note on versions. The configuration examples were written for vLLM 0.9.x, Karpenter v1 and LeaderWorkerSet v1. The screenshots come from vLLM 0.29.0, which renamed two metrics, and section 7 lists both names. Instance specs, prices and framework flags change often, so check the current AWS documentation and pricing before you commit to a design.

A note on the screenshots. My AWS account started with a quota of zero for both P and G instances, so I couldn't take these screenshots on GPUs. Instead I ran the same stack at small scale: Amazon EKS, Karpenter, vLLM, Prometheus, Grafana and KEDA on CPU nodes (c7i.xlarge), serving Qwen2.5-0.5B-Instruct. Every screenshot comes from that real cluster. What you'll see in them (concurrency limited by the KV cache, autoscaling on queue depth, the benchmark knee) works the same way on GPUs, but the absolute numbers don't carry over.

Figure 1: The reference architecture this article builds toward. — Figure 1: Reference architecture, with ALB and gateway routing into vLLM replicas on EKS, connected by EFA

1. Do the memory math first

Before picking an instance type or a parallelism strategy, you need three numbers: how much memory the weights take, how much the KV cache takes per token, and how much headroom the runtime needs.

What the KV cache is

While a model generates text, it keeps a small block of data for every token it has already processed, so it doesn't have to redo that work on each step. This store is the KV cache (K and V are the "key" and "value" tensors from attention).

It grows with every token of every conversation currently in flight, and it lives in GPU memory next to the weights. That's why it matters so much. The weights decide whether the model loads at all, while the KV cache decides how many people you can serve at the same time.

Weights

Multiply the parameter count by the bytes per parameter.

Precision	Bytes per parameter	70B model
FP16 / BF16	2	~140 GB
FP8	1	~70 GB
INT4 / AWQ	~0.5	~35 GB

KV cache per token

1
2
3
4
5
bytes_per_token = 2                    (one for K, one for V)

                x num_layers

                x num_kv_heads

                x head_dim

                x bytes_per_element

Llama 3.1 70B has 80 layers, 8 KV heads (it uses grouped-query attention, which is why it's 8 and not 64), a head dimension of 128, and 2 bytes per element in FP16:

1
2 x 80 x 8 x 128 x 2 = 327,680 bytes, or about 320 KB per token

To put that in perspective, one 8,000-token conversation needs roughly 2.5 GB of KV cache. A single 128k-context request needs around 40 GB, which is half an H100 for one user.

Headroom

Activations, CUDA graphs, memory fragmentation and the framework itself all need space. I budget 10 to 15 percent for them.

One p5.48xlarge, worked through

A p5.48xlarge has 8 x 80 GB = 640 GB of GPU memory:

Item	FP16	FP8
Weights	140 GB	70 GB
Overhead (about 12%)	~75 GB	~75 GB
Left for KV cache	~425 GB	~495 GB
Approx. concurrent 8k-token conversations	~170	~198

This table is the reason to use eight GPUs. The model fits on two H100s, but it can serve far more users on eight, because the extra memory goes to the KV cache. More cache means more concurrent requests, and that's where throughput comes from.

The rule of thumb I rely on is to size for the KV cache rather than the weights. If KV cache utilization sits at 100 percent while requests are queuing, the replica is short on memory, not compute. The fix is more replicas or quantization, and a faster GPU won't help.

2. Picking a parallelism strategy

There are four ways to spread an LLM over multiple GPUs. They solve different problems, and picking the wrong one can make a deployment slower instead of faster.

Tensor parallelism (TP)

Tensor parallelism splits every matrix multiplication across GPUs, so each GPU computes one slice of the same operation. After every layer the GPUs combine their partial results with a collective operation called an all-reduce. Because that happens constantly, the speed of the link between GPUs matters a lot.

Keep TP inside a single machine with a fast GPU-to-GPU interconnect. TP=8 over NVSwitch on a P5 works very well. TP=8 over PCIe on a G6e is noticeably slower. TP across machines, over the network, is almost always a mistake.

Pipeline parallelism (PP)

Pipeline parallelism splits the model into groups of layers, so machine A might run layers 1 to 40 and machine B layers 41 to 80. The only thing sent between stages is one activation tensor, which is small, point to point, and tolerant of network latency. The price is pipeline bubbles, where a stage sits idle waiting for work. Bubbles hurt at small batch sizes and matter much less at large ones. PP is the one to stretch across machines.

Replicas (data parallelism)

With replicas, each machine runs a complete copy of the model and a router spreads traffic between them. Replicas don't talk to each other at all, so the interconnect doesn't matter. You get the best throughput per dollar and the best failure isolation, so use replicas whenever the model fits in one machine.

Expert parallelism (EP)

EP only applies to mixture-of-experts models, where the experts are spread across devices. The bottleneck (all-to-all routing) and the tuning are different, so I don't cover it here.

Deciding

1
2
3
4
5
Does the model plus a useful KV cache fit on one GPU?

  yes -> single GPU, scale with replicas. Stop.

Does it fit on one machine?

  yes -> TP = GPUs per machine, scale with replicas. Stop.

  no  -> TP within the machine, PP across machines, then scale that unit with replicas.

70B in FP16 fits on one machine, so the production layout is TP=8 with one replica per p5.48xlarge, and as many replicas behind a router as the load needs. Section 5 still covers the multi-machine setup, because 405B-class models and very long context windows need it.

Check your accelerated-instance quota in the Service Quotas console before you build anything. "Running On-Demand P instances" and "Running On-Demand G and VT instances" are measured in vCPUs, a new account usually starts at zero for both, and an increase for P instances can take days. My request went to a support case.

3. Getting 140 GB of weights onto a new node

Loading 140 GB of weights turns into a real problem once Karpenter starts adding and removing nodes, because every new node has to fetch that data before it can serve its first request.

These are the four options, roughly in the order I'd pick them:

Amazon S3 with a streaming loader. Keep the weights in S3 and load them directly with vLLM's --load-format runai_streamer. It's fast, needs no extra infrastructure, and scales with S3 throughput.
Amazon FSx for Lustre linked to the S3 bucket. This gives the best sustained throughput when many pods load at once, but you now have a filesystem to run.
Mountpoint for Amazon S3 CSI driver. It's simple and close to POSIX, and a good fit when each pod reads the weights once at startup.
Weights baked into the container image. This works, but you end up with a 150 GB image and startup time is dominated by Amazon ECR pulls. I'd only do it for pinned models that rarely change.

Whichever you choose, put the node group in a cluster placement group so the nodes land physically close together. Also keep one small replica running at all times, so a scale-from-zero event never lands on a real user.

4. Single-node deployment on Amazon EKS (TP=8)

This assumes you already have an EKS cluster with the NVIDIA device plugin installed, and Karpenter (or a managed node group) that can launch p5.48xlarge instances.

Start with a Karpenter NodePool limited to that instance type:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
apiVersion: karpenter.sh/v1

kind: NodePool

metadata:

  name: gpu-inference

spec:

  template:

    spec:

      requirements:

        - key: node.kubernetes.io/instance-type

          operator: In

          values: ["p5.48xlarge"]

        - key: karpenter.sh/capacity-type

          operator: In

          values: ["on-demand"]

      taints:

        - key: nvidia.com/gpu

          effect: NoSchedule

      nodeClassRef:

        group: karpenter.k8s.aws

        kind: EC2NodeClass

        name: gpu-inference

  limits:

    nvidia.com/gpu: 64

Then the Deployment that runs vLLM:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
apiVersion: apps/v1

kind: Deployment

metadata:

  name: vllm-llama70b

spec:

  replicas: 2

  selector:

    matchLabels: { app: vllm-llama70b }

  template:

    metadata:

      labels: { app: vllm-llama70b }

    spec:

      tolerations:

        - key: nvidia.com/gpu

          effect: NoSchedule

      containers:

        - name: vllm

          image: vllm/vllm-openai:latest   # pin a real version in production

          args:

            - "--model=meta-llama/Llama-3.1-70B-Instruct"

            - "--tensor-parallel-size=8"

            - "--max-model-len=16384"

            - "--gpu-memory-utilization=0.92"

            - "--enable-prefix-caching"

            - "--enable-chunked-prefill"

          ports:

            - containerPort: 8000

          resources:

            limits:

              nvidia.com/gpu: 8

          readinessProbe:

            httpGet: { path: /health, port: 8000 }

            initialDelaySeconds: 180

            periodSeconds: 10

          volumeMounts:

            - { name: shm, mountPath: /dev/shm }

      volumes:

        - name: shm

          emptyDir: { medium: Memory, sizeLimit: 32Gi }

Settings worth understanding

--gpu-memory-utilization=0.92 sets how much GPU memory vLLM takes for the weights and KV cache. Raise it until you see out-of-memory errors under load, then step back a little. Going from 0.85 to 0.93 frees tens of gigabytes for the KV cache, which from section 1 means tens more users at once.

--enable-prefix-caching reuses cached blocks when requests start with the same text. If every request carries the same long system prompt or the same retrieved document, that shared part is computed only once. On chat and retrieval workloads this often cuts time to first token by a lot, and it costs nothing.

--enable-chunked-prefill lets vLLM mix prompt processing with ongoing generation, so one very long prompt can't stall every response already in progress. If p99 latency is terrible while the average looks fine, this is the first thing I'd turn on.

The /dev/shm mount is required. Tensor-parallel workers communicate through shared memory, and the default 64 MB container limit breaks them.

Check the topology before trusting a benchmark

1
kubectl exec -it deploy/vllm-llama70b -- nvidia-smi topo -m

Every pair of GPUs should show NV#, which means a direct NVLink connection. If you see PHB or SYS inside the tensor-parallel group, traffic is going over PCIe or through the CPU, and it will be slow.

Next, read the startup log. vLLM reports how much KV cache it allocated and how many concurrent requests that supports.

Figure 2: Checking the section 1 formula against a real log. For Qwen2.5-0.5B (24 layers, 2 KV heads, head dimension 64, bfloat16), the formula gives 2 x 24 x 2 x 64 x 2 = 12,288 bytes per token, so a 2 GiB budget should hold about 174,760 tokens. vLLM allocated 174,720, since it rounds down to whole cache blocks, and reports 42.66x concurrency at 4,096 tokens. The log calls it a GPU KV cache even on a CPU node. On GPUs, if the number is far below your estimate, check `--gpu-memory-utilization` first. — Figure 2: vLLM 0.29 startup log from the demo cluster, filtered to the important lines. It shows 2.0 GiB reserved for the KV cache, a KV cache size of 174,720 tokens, and a maximum concurrency of 42.66x for 4,096-token requests

5. Going multi-node

Skip this section if your model fits on one machine. Most 70B deployments do, and running a model across machines adds a lot of operational work. You need it for 405B-class models and for very long context windows.

Once a model spans machines, network speed directly limits how fast it runs.

NCCL and EFA

NCCL is the library GPUs use to talk to each other. vLLM doesn't use the network directly, it goes through NCCL.

EFA (Elastic Fabric Adapter) is the AWS network interface that lets NCCL bypass the operating system's network stack, with up to 3,200 Gbps on P5-class instances. Without EFA, the collectives run over ordinary TCP, and multi-node inference ends up slower than just using one machine.

What has to be in place

All three of these need to be true:

The instances support EFA and sit in a cluster placement group.
The EFA Kubernetes device plugin is installed, so pods can request the vpc.amazonaws.com/efa resource.
The container includes the aws-ofi-nccl plugin, so NCCL uses EFA instead of quietly falling back to sockets.

1
2
3
4
5
6
7
8
resources:

  limits:

    nvidia.com/gpu: 8

    vpc.amazonaws.com/efa: 32   # device count is instance-type specific

env:

  - { name: FI_PROVIDER,     value: "efa" }

  - { name: NCCL_DEBUG,      value: "INFO" }

  - { name: LD_LIBRARY_PATH, value: "/opt/amazon/ofi-nccl/lib:/opt/amazon/efa/lib:$(LD_LIBRARY_PATH)" }

On the first deploy, set NCCL_DEBUG=INFO and read the logs. If you don't see the OFI or EFA provider being selected, NCCL is running over TCP, and you've found your performance problem.

Use LeaderWorkerSet instead of a Deployment

Multi-node vLLM has to be scheduled as a group. There's one leader and N workers, and if any of them is missing the whole group is useless. A Deployment doesn't understand that and will happily run one pod while the other is stuck pending. The LeaderWorkerSet (LWS) API does, and it restarts the whole group when one pod fails.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
apiVersion: leaderworkerset.x-k8s.io/v1

kind: LeaderWorkerSet

metadata:

  name: vllm-multinode

spec:

  replicas: 1

  leaderWorkerTemplate:

    size: 2                      # 1 leader + 1 worker = 16 GPUs total

    restartPolicy: RecreateGroupOnPodRestart

    leaderTemplate:

      spec:

        containers:

          - name: vllm-leader

            image: vllm/vllm-openai:latest

            command: ["/bin/sh", "-c"]

            args:

              - |

                ray start --head --port=6379 --block &

                vllm serve meta-llama/Llama-3.1-405B-Instruct-FP8 \

                  --tensor-parallel-size 8 \

                  --pipeline-parallel-size 2 \

                  --max-model-len 32768

            resources:

              limits: { nvidia.com/gpu: 8, vpc.amazonaws.com/efa: 32 }

    workerTemplate:

      spec:

        containers:

          - name: vllm-worker

            image: vllm/vllm-openai:latest

            command: ["/bin/sh", "-c"]

            args: ["ray start --address=$(LWS_LEADER_ADDRESS):6379 --block"]

            resources:

              limits: { nvidia.com/gpu: 8, vpc.amazonaws.com/efa: 32 }

This is the section 2 rule written as YAML, with TP=8 inside each machine and PP=2 across the two machines.

Test the network before you start blaming the model:

1
mpirun -n 16 -N 8 /opt/nccl-tests/build/all_reduce_perf -b 1M -e 8G -f 2 -g 1

There are two things to check. The leader and worker pods should both show 1/1 Running, on different nodes. If a worker is stuck in Pending, look at the EFA resource counts and placement group capacity. And at large message sizes, the NCCL bus bandwidth should get close to the instance's rated network throughput. If it levels off far below that, NCCL isn't using EFA.

6. Routing requests between replicas

Round-robin load balancing works poorly for LLM traffic, because requests vary so much in size. One request might generate 8 tokens and the next one 4,000. Spreading requests evenly by count doesn't spread the work evenly.

Two approaches work better.

The first is to route on queue depth rather than connection count. The number of requests already waiting at a replica is the best predictor of how long the next one will take.

The second is to route on shared prefixes. Requests that start the same way (the same system prompt, the same retrieved document, the same conversation) should go to the replica that already has those blocks in its cache. On chat and retrieval workloads this is often the biggest improvement you can make, because a cache miss turns into a cache hit.

In Kubernetes, the current way to do this is the Gateway API Inference Extension. It adds an InferencePool and an endpoint picker that routes based on live vLLM metrics, behind an Application Load Balancer or an in-cluster gateway.

If you want something simpler to start with, a Service using least_request load balancing with sticky sessions keyed on conversation ID gets you a good part of the benefit for much less work.

Keep in mind that a plain Kubernetes Service balances per connection, not per request. I ran into this during my benchmark runs. KEDA added two replicas in the middle of a sweep, and Prometheus showed the new pods serving no requests at all for about three minutes while the first pod had a queue. The benchmark client had opened its connections before the new pods existed and kept reusing them. Any client with long-lived connections behaves the same way in production, which is one more reason to route per request.

A few practical points. Terminate TLS at the ALB, authenticate at the gateway, and raise the idle timeout. Streaming responses keep connections open for a long time, and the default 60-second idle timeout will cut them off mid-answer.

7. Autoscaling

Don't autoscale LLM serving on CPU utilization, and be careful with GPU utilization as well. A GPU that's waiting on memory during decoding can report high utilization while doing very little useful work.

Scale on queue depth and cache pressure instead. vLLM exports both on /metrics:

Metric	What it means	Use it for
`vllm:num_requests_waiting`	Requests queued but not started yet	Main scale-out trigger
`vllm:num_requests_running`	Requests being processed right now	Checking saturation
`vllm:kv_cache_usage_perc`	How full the KV cache is	Spotting a replica that's short on memory
`vllm:time_to_first_token_seconds`	How long a user waits before the first word appears (TTFT)	User-facing latency SLO
`vllm:request_time_per_output_token_seconds`	Gap between words once streaming starts (TPOT)	Streaming smoothness SLO

vLLM renamed two of these in newer releases. KV cache usage used to be vllm:gpu_cache_usage_perc, and TPOT used to be vllm:time_per_output_token_seconds. A dashboard that only queries one set of names shows "No data" on the other version, which would have left two of my four Grafana panels empty. Querying both avoids that, for example avg(vllm:kv_cache_usage_perc) or avg(vllm:gpu_cache_usage_perc).

Here's a KEDA ScaledObject that scales on the queue:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
apiVersion: keda.sh/v1alpha1

kind: ScaledObject

metadata:

  name: vllm-scaler

spec:

  scaleTargetRef: { name: vllm-llama70b }

  minReplicaCount: 1

  maxReplicaCount: 8

  cooldownPeriod: 600

  triggers:

    - type: prometheus

      metadata:

        serverAddress: http://prometheus.monitoring:9090

        query: sum(vllm:num_requests_waiting)

        threshold: "5"

Two settings matter more than the threshold itself. Set a long cooldown, ten minutes or more, because a new GPU node plus a 140 GB weight load takes minutes, and scaling up and down repeatedly gets expensive. Also keep minReplicaCount at 1 or higher, unless you can live with a first request that takes several minutes.

Figure 3: The four panels I use to diagnose a serving problem, recorded during a load test. The queue reached 24 requests while KV cache utilization never went above 1.8%, so this replica was short on compute, not memory. Going by the section 1 rule of thumb, the fix is more replicas, and that's what KEDA did next. — Figure 3: Grafana dashboard from the demo cluster over 30 minutes. P99 time to first token climbs to about 1.3 minutes during the load test and drops after scale-out. P99 time per output token stays between 150 and 165 milliseconds. Waiting requests peak at 24, and KV cache utilization stays close to zero

Figure 4: A scale-out on the demo cluster. With 24 requests queued against a target of 5 (top), KEDA raised the deployment from 1 to 3 replicas. The `5334m/5` value is Kubernetes notation for 5.33. The new pods stayed in `Pending` for about 30 seconds while Karpenter launched nodes, then moved to `ContainerCreating` (bottom). Each new replica took about 3 to 4 minutes to become ready. That delay is your cold-start budget, and it should be part of your SLO. — Figure 4: Two terminal windows from the demo cluster. In the top window, the KEDA HorizontalPodAutoscaler target rises from 0/5 to 24/5 and replicas go from 1 to 3. In the bottom window, a pod watch shows two new vLLM pods in Pending until Karpenter assigns them nodes, then ContainerCreating

Figure 5: Three independent replicas on three separate nodes, which is the layout section 2 recommends when a model fits on one machine. The two new pods show `0/1 Running` because vLLM is still loading the model, and they switch to `1/1` once the readiness probe passes. — Figure 5: kubectl get pods output showing three vLLM replicas, each on a different node. The original pod is 1/1 Running, and the two new pods are Running but not ready yet

8. Benchmarking

A latency number from a single request tells you very little. What you want is a sweep, where you run the same workload at increasing concurrency and watch throughput and latency together.

1
2
3
4
5
6
7
8
vllm bench serve \

  --backend openai \

  --model meta-llama/Llama-3.1-70B-Instruct \

  --dataset-name random \

  --random-input-len 2048 \

  --random-output-len 256 \

  --max-concurrency 64 \

  --num-prompts 1000

When you report results, always include these four together:

Output tokens per second across the whole system
TTFT at p50 and p99
TPOT at p50 and p99
The concurrency they were measured at

Any one of them on its own says almost nothing about how the system will behave under real load.

Figure 6: Benchmark summary from the demo cluster, with Qwen2.5-0.5B on one `c7i.xlarge`, 256-token prompts, 128-token outputs and a concurrency of 8. Capture the whole block. The percentiles matter most, because averages hide the slow requests that users notice. — Figure 6: vllm bench serve summary from the demo cluster at concurrency 8 with 100 requests. Output token throughput is 59.61 tokens per second, median TTFT 886 milliseconds, P99 TTFT 1,017 milliseconds, median TPOT 122.9 milliseconds and P99 TPOT 132.9 milliseconds

Finding the knee

Run the benchmark at several concurrency levels and plot the results. You'll almost always see the same shape. Throughput rises quickly and then levels off, while latency stays flat for a while and then climbs sharply. The point where that happens is the knee, and it's the concurrency you should run at.

Beyond the knee you pay a lot of latency for very little extra throughput. Set --max-num-seqs close to the knee, so vLLM queues extra requests instead of accepting work it can't serve well.

Figure 7: A concurrency sweep on the CPU demo cluster (Qwen2.5-0.5B on c7i.xlarge, vLLM 0.29). Throughput grows almost linearly up to `--max-num-seqs=8` and then stops, matching Figure 6 at that point. Past it, time per output token hardly changes, but time to first token goes from 0.9 s to 52 s. The extra requests wait in the queue instead of slowing generation down. GPUs show the same shape, with very different numbers. — Figure 7: Concurrency sweep on the CPU demo cluster. On the left, output throughput rises to about 62 tokens per second at concurrency 8 and then stays flat. On the right, on a log scale, median time to first token jumps from under a second to 52 seconds past the knee, while median time per output token stays between 110 and 126 milliseconds

9. Cost model

To compare designs, convert everything into cost per million output tokens:

1
cost_per_1M_output_tokens = (instance_hourly_rate / (output_tokens_per_sec x 3600)) x 1,000,000

Use the throughput you measured at the knee over a sustained run, and the current instance price from the EC2 pricing page.

For example, if your sweep shows 2,000 output tokens per second and the instance costs $R per hour, you pay about $R x 0.139 per million output tokens. At a hypothetical $50 per hour, that's about $6.95 per million.

These are the levers I'd work through, starting with the one that usually saves the most:

Raise utilization. An instance running at 25 percent utilization costs four times as much per token. Better batching, smarter routing and sharing replicas between models save more than any low-level tuning.
Quantize. FP8 on H100-class hardware roughly halves the memory the weights need, and that memory goes straight to KV cache and concurrency. Check quality on your own evaluations rather than a public leaderboard.
Commit to capacity. Savings Plans or Reserved Instances suit steady load, EC2 Capacity Blocks for ML suit scheduled batch windows, and Spot suits offline inference that can tolerate interruptions.
Right-size the accelerator. For models that fit in 48 GB, G6e (NVIDIA L40S) often gives much better price-performance than P5. Inf2 and Trn instances with AWS Neuron can be cheaper still for supported models, at the cost of a compilation step and a smaller model catalog.
Cut wasted tokens. Prefix caching, shorter prompts and stop sequences all reduce work before it reaches the GPU.

Tag your inference nodes, for example with workload=llm-inference, and activate that tag as a cost allocation tag in the Billing console. Cost Explorer can then break spend down by model and environment instead of lumping it into one EC2 line. Activation doesn't apply retroactively, so do it before the cluster starts running.

10. Common failure modes

Symptom	Likely cause	Fix
Tensor-parallel workers hang or crash at startup without a clear error	`/dev/shm` left at the 64 MB container default	Mount a memory-backed `emptyDir` of 16 to 32 GiB at `/dev/shm`
Multi-node throughput is a fraction of what you expected	NCCL fell back to TCP instead of using EFA	Set `NCCL_DEBUG=INFO` on first boot and confirm the EFA/OFI provider is selected
Concurrency is far below your section 1 estimate	`gpu-memory-utilization` left at its default	Raise it to between 0.90 and 0.95, then check the KV cache line in the startup log again
p99 latency is terrible while p50 looks healthy	Chunked prefill is off, so one long prompt stalls every stream	Add `--enable-chunked-prefill`
Scaling happens at the wrong times	Autoscaling on GPU utilization	Scale on `vllm:num_requests_waiting` instead
Pods keep getting killed while loading weights	Readiness probe is too aggressive	Set `initialDelaySeconds` above your measured load time
New replicas are ready but idle while the first one is overloaded	Long-lived connections stay on the old pod, because a Service balances per connection	Route per request, with the Gateway API Inference Extension, Envoy or an ALB
Latency varies in a way no tuning fixes	No cluster placement group	Put the GPU node group in one
p99 alerts that wake you at 3 a.m.	Cold start isn't part of the SLO	Keep `minReplicaCount` at 1 or more, and budget for scale-from-zero

11. When not to build this

Running distributed inference yourself takes real operational effort, so make sure you need it first. Here are the alternatives, roughly from least to most work.

If a hosted model on Amazon Bedrock meets your quality bar, you can skip everything in this article. I'd start there and only move off it for a clear reason.

Amazon SageMaker AI real-time endpoints with the Large Model Inference (LMI) containers run vLLM under the hood and handle autoscaling, deployment and rollout for you. They're a good middle ground when you need custom weights but not custom infrastructure.

Amazon SageMaker HyperPod fits large, long-running GPU clusters where you want orchestration that copes with node failures, for both training and inference.

Amazon EKS, as described here, makes sense when you need control. That could be custom routing, running next to the rest of your platform, packing several models onto shared hardware, or a serving layout the managed options don't support.

One thing I'm keeping an eye on is prefill/decode disaggregation. Prefill is limited by compute and decode is limited by memory, so the idea is to run the two phases on separately scaled pools and move the KV cache between them over the network. With EFA-class networking this is becoming practical.

Clean up

GPU instances are the most expensive thing to forget about.

1
2
3
4
kubectl delete -f vllm-deployment.yaml

kubectl delete nodepool gpu-inference

# then confirm the nodes are actually gone

kubectl get nodes -l karpenter.sh/nodepool=gpu-inference

After that, check in the EC2 console that no accelerated instances are still running, and delete any FSx for Lustre filesystem you created. I trust the console over the terminal here, because a pod can be gone while the node it ran on keeps billing.

Terms used in this article

Term	Meaning
KV cache	State the model keeps for tokens it has already processed. It lives in GPU memory and limits concurrency.
TTFT	Time to first token, the time a user waits before any output appears.
TPOT	Time per output token, the gap between tokens once streaming has started.
TP / PP	Tensor parallelism (each operation split across GPUs) and pipeline parallelism (the model split by layers).
NCCL	The library GPUs use to communicate with each other.
EFA	Elastic Fabric Adapter, the AWS networking that lets NCCL bypass the operating system's network stack.
All-reduce	A collective operation in which every GPU combines its partial result with those of all the others.
Gang scheduling	Scheduling a group of pods as one unit, so they all start together or none of them do.
Prefix caching	Reusing cached KV blocks across requests that start with the same text.
Chunked prefill	Mixing prompt processing with ongoing generation, so long prompts don't stall other streams.

Summary

Most of distributed inference comes down to matching how the parts of your model communicate with the network you have, and then measuring the right things.

Size for the KV cache rather than the weights. It sets concurrency, and concurrency sets throughput.
Use TP inside a machine, PP across machines, and replicas for everything else.
Multi-node needs EFA and aws-ofi-nccl, and you should confirm NCCL is using them rather than assume it.
Route on queue depth and shared prefixes, and autoscale on num_requests_waiting, never on CPU.
Report throughput and latency percentiles together, at a stated concurrency.
Track cost per million tokens. Utilization affects it more than anything else.

Versions, instance specs and prices change often, so check the current AWS documentation and pricing before you commit to a design. If you've measured your own numbers, I'd like to hear about them in the comments. The interesting differences usually come from the shape of the workload.

# generative-ai # amazon-eks # amazon-ec2 # machine-learning # containers

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Distributed LLM Inference on AWS: Serving a 70B Model Across Multi-GPU and Multi-Node with vLLM on EKS

Mohammad AlYasfo

1. Do the memory math first

What the KV cache is

Weights

KV cache per token

Headroom

One p5.48xlarge, worked through

2. Picking a parallelism strategy

Tensor parallelism (TP)

Pipeline parallelism (PP)

Replicas (data parallelism)

Expert parallelism (EP)

Deciding

3. Getting 140 GB of weights onto a new node

4. Single-node deployment on Amazon EKS (TP=8)

Settings worth understanding

Check the topology before trusting a benchmark

5. Going multi-node

NCCL and EFA

What has to be in place

Use LeaderWorkerSet instead of a Deployment

6. Routing requests between replicas

7. Autoscaling

8. Benchmarking

Finding the knee

9. Cost model

10. Common failure modes

11. When not to build this

Clean up

Terms used in this article

Summary