AWS Builder Center

Routing LLM traffic across GPU clusters for throughput, with llm-d

GPU capacity is often distributed, so we run llm-d in many clusters. Requests can land in the wrong one. We demonstrate a global llm-d hub on Amazon EKS across three AWS regions. We wrote a small plugin that routes on inter-token latency. It fills the fastest cluster to its batch limit, then spills to the next. On real vLLM it gave 13% more tokens per second than upstream llm-d, with half the time to first token. We cover what we learned, what worked, what didn't, and how to deploy it.

GPU capacity is globally distributed. You take what each region will give you. So you run several inference clusters on AWS, some on EKS and some on SageMaker HyperPod. Each one runs llm-d. Inside a cluster, llm-d's endpoint picker (EPP) does a good job. It sends each request to a good pod.
Across clusters, routing becomes more difficult. Requests often land in the wrong cluster, and then on a poor pod in that cluster. When you care about throughput above all, inter-token latency (ITL) matters more than time to first token (TTFT).
This post covers one day of building a fix. We stood up four EKS clusters in three regions and tried several routing policies, first on a simulator and then on real GPUs. Some of our ideas failed, and the failures taught us the most. So we recount them here, for you. The result is a small llm-d plugin. At 160 concurrent streams on real vLLM it delivers 13% more tokens per second than the best upstream policy, with half the TTFT.

The first idea

The plan was simple. Put a global entrypoint cluster in front of everything, and run llm-d there too.
  1. Each inference cluster registers with the entrypoint when it starts and leaves when it stops.
  1. Each cluster's latency sidecar writes its best pod's ITL into Redis in the entrypoint region.
  1. A custom sidecar in the entrypoint reads Redis and returns the cluster with the lowest ITL.
  1. The global llm-d sends the request there, and that cluster's llm-d picks the pod.
Before building it, we read the llm-d and Gateway API Inference Extension docs and code. Three parts of the plan would not have worked as written.
  • An InferencePool only selects Pods. It matches Pods by label in its own namespace. It can't point at a Service, an ExternalName or an IP outside the cluster. So "show up as an external service" was out.
  • The latency predictor doesn't pick endpoints. The EPP sends the predictor sidecar a feature vector for each candidate pod, such as KV cache use, queue depth and input length. The sidecar sends back a predicted TTFT and TPOT. A scorer plugin inside the EPP does the ranking. The feature vector doesn't say which endpoint it describes, so a custom sidecar can't map a Redis value back to a cluster. The choice has to happen in a plugin, written in Go and compiled into the EPP.
  • A proxy has no metrics. The EPP scrapes vLLM metrics from every endpoint. A plain proxy has none, so the EPP treats it as saturated.
We also suspected the signal itself. The best pod's ITL is one snapshot. It says nothing about spare capacity. If every request chases the lowest number, they all go to one cluster. We kept the idea in the test anyway. We wanted to see it fail.

llm-d already has a hub mode

llm-d-router v0.11.0 ships Alpha "hub" plugins. A hub EPP treats whole clusters as its endpoints.
  •  multicluster-file-discovery  reads a list of clusters from a file and reloads it when the file changes.
  •  multicluster-metrics-data-source  scrapes each leaf cluster's EPP for pool averages, such as KV cache use and queue depth.
  • Two scoring policies are shipped.  load  ranks clusters on those pool averages.  latency  learns each cluster's TTFT from the responses the hub sees.
Nothing in hub mode scores on ITL. The leaf EPP records an ITL histogram, but the hub's metrics extractor reads gauges and counters only. So for throughput routing, we had to write something.

Why not route on the TPOT llm-d already exports?

The leaf EPP exports time per output token (TPOT) as well as ITL. The two are close, but they differ.
  •  llm_d_epp_request_streaming_tpot_seconds  gives one number per request. It is (end-to-end time − TTFT) ÷ (output tokens − 1), the request's average decode speed.
  •  llm_d_epp_request_streaming_itl_seconds  records every gap between streamed chunks. It shows the spikes, such as a new request's prefill stalling the whole batch.
The averages come out about the same. The tails differ. A request can have a fine TPOT and still contain a few 150 ms stalls. We saw exactly that on a T4 later in this post.
So why doesn't llm-d route on a metric it exports? Because measured TPOT lags. It tells you what the last requests got, after the fact. Send traffic to whoever had the best recent number, and every request goes to the same place until that number gets worse. Our first idea failed this way.
llm-d predicts TPOT instead. Inside a cluster, predicted latency routing sends each pod's live load to a predictor. The load includes KV cache use, queue depth, running requests and prefix match. The predictor returns a TTFT and a TPOT, and llm-d scores on both. By default it weights TTFT 0.8 and TPOT 0.2. A request can set a TPOT target with the  x-llm-d-slo-tpot-ms  header. Measured TPOT is the predictor's training data.
That design assumes every pod in a pool is the same. The docs say "heterogeneous pools are not yet modeled". With the same GPU and the same model, TPOT differences come mostly from batch size, and the load scorers already see batch size. TTFT varies more, because of queueing and prefix cache hits, so it gets the bigger weight.
Across clusters that assumption breaks. Different GPUs give different TPOT for reasons that load can't explain. Our plugin fills that gap the same way llm-d does inside a cluster. It predicts from load, and it doesn't react to past numbers. The prediction is a fitted ITL-versus-load line per cluster, used with the hub's live count of requests in flight.
There was one more gap. The hub's Envoy forwards to  ip:port  only. AWS load balancers publish host names. The upstream guide says AWS is "not supported yet". We worked around it. An internal NLB keeps its private IPs for its whole life, so we resolve the name once and register the IP.

The test bed

Cluster
Region
Role
gr-hub
us-east-2
Entrypoint. Hub EPP, registrar, load generator
gr-leaf-use2
us-east-2
Leaf, 2.6 ms from the hub
gr-leaf-use1
us-east-1
Leaf, 18 ms from the hub
gr-leaf-euw2
eu-west-2
Leaf, 93 ms from the hub
Each cluster has its own VPC with a unique /16 from one /14 block. A Transit Gateway in each region is peered with the one in us-east-2. TGW peering needs static routes on both sides, so adding a region is a script step. One route for the whole /14 means each VPC route table needs one entry, however many leaves join.
Each leaf runs the llm-d router with an internal NLB in front.
We started with the llm-d inference simulator in place of vLLM. It let us set each cluster's speed on purpose. Idle ITL was 15 ms in us-east-2, 25 ms in us-east-1 and 40 ms in eu-west-2. ITL doubled at a full batch of 16. Later we swapped in real vLLM on GPUs.

The registry

We kept the Redis idea, as a registry. The store is ElastiCache for Valkey in the hub region, with TLS on. Each leaf runs a small publisher. Every 2 seconds it reads its own leaf's metrics and writes one record to  llmd:cluster:<name>  with a 10 second expiry.
  • It reads the leaf EPP for ready pods, queue depth, and the streaming ITL histogram.
  • It reads every model server pod for that pod's ITL and running requests.
  • It fits a straight line, ITL = a + b × running requests per pod, over the last 300 per-pod samples.
  • It resolves its own NLB name to an IP for the hub.
  • If its cluster has been idle for 15 seconds, it sends one small request through its own router. An idle cluster has no fresh ITL, and a cluster with no ITL never gets picked.
On the hub, a registrar turns the live records into the hub's cluster file each second. When the publishers started, the hub's list grew from one cluster to three with no hand edits.

The plugin

 cluster-itl-scorer  is about 250 lines of Go in a copy of llm-d-router v0.11.0. It polls Valkey every 500 ms. It plays two roles.
  • Filter. It drops a cluster whose record is missing or more than 10 seconds old. That takes under a second. Kubernetes can take up to a minute to push a changed ConfigMap into a pod, so the cluster file alone is slow.
  • Scorer. It ranks clusters by the ITL a new request should expect. Version one had two modes.  best-pod  used the best pod's ITL, which was our original idea.  predicted  used the fitted line at the load the cluster would have after this request. It counts that load from the hub's own live count of requests in flight.
We built it in CodeBuild. Source went to S3,  go test  ran in a golang container, and the image went to ECR.

Round one on the simulator

At 32 streams the ITL policies won on every throughput number. They delivered 30% more tokens per second than upstream  load , and ITL p90 was 28 ms against 50 ms.  load  spread traffic evenly, including to the cluster that decodes 2.7 times slower.
At 96 streams both ITL policies showed herd behavior. Every request saw the same "best" cluster and went there, until that cluster overloaded. Best-pod sent 85% of requests to us-east-2. Its 48 decode slots overflowed, TTFT p50 rose to 1.9 seconds, and throughput fell below  load . A fresh low ITL pulls traffic until the next 2 second publish shows the damage.
 predicted  herded too, and it shouldn't have. We had listed the hub's in-flight count as an optional input. The EPP's loader only creates a default data producer for required inputs. So the count was always zero, and the plugin was blind to its own decisions. A one-word fix and a test fixed it.
The fixed  predicted  mode was better, but TTFT was still high. The fitted line said the fast cluster stayed the cheapest well past its batch limit, so requests queued there. A queued request has a fine ITL once it runs. It waits first. So we added a weight for queueing. With  queuePenalty: 10 , the plugin fills the nfastest cluster up to its batch limit and then moves to the next one. We call it spill.
At 96 streams spill gave 2,010 tokens/s, the best of any policy, against 1,824 for  load  and 1,900 for  latency . TTFT p50 fell from 888 ms to 320 ms.

Breaking things on purpose

We killed a leaf under load to see what the hub does.
Clean shutdown. The publisher deletes its key, then the router stops. Zero errors. The filter dropped the cluster within a second, and traffic moved.
Crash. The publisher and router die together, so nobody deletes the key. This caused a full outage for about 15 seconds. The two healthy clusters were idle. The dead cluster's load balancer reset each connection at once, so its requests failed in milliseconds. Its in-flight count fell to zero. With its last record still fresh, it looked like the best cluster, and the hub sent it everything. Any load-aware scorer will read a fast failure as spare capacity.
The fix was a health check from the hub. The plugin probes each cluster every second and drops it after two failures. The outage fell to about 3 seconds. Retries would hide those last seconds. The EPP protocol can return fallback endpoints, and Envoy can retry on connect failure. We didn't build that yet.
Registry outage. We rebooted the Valkey node under load. Zero errors, and throughput stayed flat. The plugin kept its last records, and when they went stale, its filter kept every cluster. A dead registry makes routing worse. It doesn't stop it.
The reboot also flushed the data. As publishers came back one by one, the registrar wrote a cluster file with one cluster in it for a second. Now the registrar keeps a cluster until its key has been gone for 15 seconds. This gave us a rule. Use two removal paths. The filter removes a cluster fast and can add it back fast. The file changes slowly and only needs to be right in the end.
Deploys looked like outages. Restarting a publisher dropped its cluster for 2 seconds. The old pod's exit handler deleted the key after the new pod had already written it. Now each record names its writer. On exit, a small Lua script deletes the key only if this pod still owns it. A readiness probe also waits for the new pod's first publish before Kubernetes stops the old one.

Real GPUs

We asked for three GPU types in three regions and got what was free.
Leaf
Asked for
Got
us-east-2
g6e.xlarge (L40S)
g6.xlarge (L4)
us-east-1
g5.xlarge (A10G)
g5.xlarge (A10G)
eu-west-2
g6.xlarge (L4)
g4dn.xlarge (T4), on the third try
Two regions returned  InsufficientInstanceCapacity  in every zone for 20 minutes. Node groups with a list of instance types got us through. This is the same capacity problem that makes multi-cluster routing worth doing. Our home region also ended up with a slower GPU than us-east-1, so a fixed region preference would have been wrong.
We ran Qwen2.5-1.5B-Instruct on vLLM v0.19.1, one GPU per pod, 2 pods per leaf, 32 sequences each. The leaf router and publisher needed no changes. They read real vLLM metrics the same way.
Real vLLM did not look like the simulator. For a small model, ITL barely rises with batch size. On the A10G it stayed near 13 ms from 4 to 32 running requests. What moves ITL is spikes. When a new request's prefill runs, every decode in the batch waits a step. One 2 second window on the T4 averaged 160 ms. Our straight line chased the spikes and came out wrong. Dropping samples above three times the median fixed the fit. The order of the clusters stayed clear the whole time. us-east-1 was fastest, then us-east-2, then eu-west-2.
The table shows results at 160 streams, which is 85% of capacity.
Policy
tokens/s
TTFT p50 / p90 (ms)
ITL p90 (ms)
upstream  load 
6,540
541 / 4,119
25.9
upstream  latency  (TTFT)
6,813 (mean of 3)
423 / 3,306
22.0
best-pod ITL (the first idea)
4,575
4,381 / 7,288
14.6
ITL spill
7,717 (mean of 3)
233 / 1,569
20.8
 load  and best-pod are single runs. Spill won on all three of its runs against  latency . The gain was 13% in tokens per second, with about half the TTFT.
Best-pod herded worse on GPUs than on the simulator. It sent 94% of requests to us-east-1. Its ITL figure looks great, because the requests that do run are on the fastest GPU. Its throughput was the worst of the four. ITL alone is the wrong goal. You have to count the wait.
Spill and upstream  latency  sent almost the same share of traffic to each region over a run. The gain came from timing. Spill moves the next request at the moment the fast cluster fills.  latency  learns from TTFT it has already seen, so it reacts after the queue has formed.
At light load, upstream  latency  had the best TTFT. ITL-aware routing still gave 5% more tokens per second.

How to deploy it

All code is available on  Github .
The plugin ships inside your own EPP image. The rest is the standard llm-d router Helm chart with a different image and config. Here are the steps.
1. Add the plugin to llm-d-router. Clone the release your chart uses. We used v0.11.0.
git clone --depth 1 --branch v0.11.0 https://github.com/llm-d/llm-d-router.git
Copy the plugin into  pkg/epp/framework/plugins/scheduling/scorer/clusteritl/ . Then register it in  cmd/epp/runner/runner.go . These two lines are the only upstream change.
import "github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/scheduling/scorer/clusteritl"

fwkplugin.Register(clusteritl.PluginType, fwkplugin.StabilityAlpha, clusteritl.Factory)
The Redis client and the test Redis are already in  go.mod , so the dependencies don't change.
2. Build and push the image. Use the repo's own  Dockerfile.epp .
docker run --rm -v $PWD:/w -w /w golang:1.26.8 \
go test ./pkg/epp/framework/plugins/scheduling/scorer/clusteritl/...
docker build -f Dockerfile.epp --build-arg TARGETOS=linux --build-arg TARGETARCH=amd64 \
-t <account>.dkr.ecr.us-east-2.amazonaws.com/llm-d-epp:clusteritl .
docker push <account>.dkr.ecr.us-east-2.amazonaws.com/llm-d-epp:clusteritl
CodeBuild ran these same steps. EKS nodes pull from ECR in the same account with no extra setup.
3. Set up the leaves and the registry. Each leaf runs the llm-d router with metrics auth turned off, so the hub can read them. An internal NLB exposes port 80 for traffic and port 9090 for metrics. The publisher runs beside the router. On the hub, the registrar keeps the  mc-hub-clusters  ConfigMap current. Each entry looks like this.
endpoints:
- name: leaf-use1 # must match the publisher's cluster name
address: "10.202.123.11" # an IP, since the hub's Envoy can't use a host name
port: "80"
labels: {metricsAddress: "10.202.123.11", metricsPort: "9090"}
4. Install the hub. Layer two values files on the chart.
helm upgrade --install hub oci://ghcr.io/llm-d/charts/llm-d-router-standalone --version v0.11.0 \
-n llm-d -f hub.values.yaml -f hub-itl-spill.values.yaml
 hub.values.yaml  comes from the upstream multi-cluster guide. It turns on experimental plugins and mounts the cluster list.  hub-itl-spill.values.yaml  swaps in the image and the plugin config.
router:
epp:
image:
registry: <account>.dkr.ecr.us-east-2.amazonaws.com
repository: llm-d-epp
tag: clusteritl
pluginsConfigFile: "itl.yaml"
pluginsCustomConfig:
itl.yaml: |
apiVersion: llm-d.ai/v1alpha1
kind: EndpointPickerConfig
plugins:
- type: multicluster-file-discovery
name: discovery
parameters: {path: /etc/epp/clusters/clusters.yaml, watchFile: true}
- type: cluster-itl-scorer
name: itl
parameters:
redisAddr: <valkey-endpoint>:6379
redisTLS: true
mode: predicted
queuePenalty: 10 # spill: fill the fastest cluster, then move on
staleAfter: 10s
healthInterval: 1s # hub-side liveness probe
healthFailures: 2
- type: max-score-picker
name: picker
- type: single-profile-handler
name: profile-handler
schedulingProfiles:
- name: default
plugins:
- pluginRef: itl
weight: 1 # one weighted entry makes it a filter and a scorer
- pluginRef: picker
dataLayer:
discovery:
endpoints: {pluginRef: discovery}
5. Check it works. The hub EPP log should show  auto-created default producer  for  inflight-load-producer , with  itl  as the consumer. That proves the plugin can see its own decisions. The parsed config should list  itl/cluster-itl-scorer  under  Filters . Run the EPP at verbosity 4 to log the cost it computes for each cluster.
Three things caused us problems.
  • Match the config version to the image. The v0.11.0 image only accepts  llm-d.ai/v1alpha1 . Guides on the main branch use  llm-d.ai/v1 .
  • Mark the in-flight count as required. If your plugin lists it as optional, the EPP never creates the producer, and the count stays at zero.
  • Turn on experimental plugins. Hub plugins and ours are Alpha. Without  --allow-experimental-plugins , the EPP refuses to start.
Plugin settings are read once, at startup. A change to  queuePenalty  or  mode  needs a  helm upgrade .

What we'd tell you to do

  1. Use llm-d hub mode for the global layer. Proxies and fake services aren't needed. Peer clusters are endpoints in a file, and the hub reads their routers' metrics.
  1. Route on expected ITL, and treat each cluster's batch limit as a soft wall. Fill the fastest cluster, then spill. Never queue on a fast cluster while a slower one has free decode slots.
  1. Use the hub's own in-flight count. Leaf metrics are seconds old. The hub knows what it sent a millisecond ago. Mark it required in your plugin.
  1. Keep Redis, as a registry with a short expiry. Add owner checks on delete and a grace period on removal.
  1. Check liveness from the hub. A dead cluster looks idle. Don't wait for a key to expire.
  1. Fit the ITL model robustly. Real ITL is spiky. A two-number line per cluster is enough if you drop the spikes.
  1. Plan CIDRs and quotas per region on day one.

What's left

  • Retries with fallback endpoints, to hide the last seconds of a crash.
  • TLS between hub and leaves. Today hub mode speaks plain HTTP over the private network.
  • A HyperPod leaf. HyperPod's Inference Gateway already runs an llm-d EPP, so it should join the same way.
  • A bigger model, where batch size moves ITL more and the fitted line matters more.
  • Upstream. The plugin is small, and an ITL scorer for hub mode would help anyone serving throughput-bound traffic.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article