Routing LLM traffic across GPU clusters for throughput, with llm-d
GPU capacity is often distributed, so we run llm-d in many clusters. Requests can land in the wrong one. We demonstrate a global llm-d hub on Amazon EKS across three AWS regions. We wrote a small plugin that routes on inter-token latency. It fills the fastest cluster to its batch limit, then spills to the next. On real vLLM it gave 13% more tokens per second than upstream llm-d, with half the time to first token. We cover what we learned, what worked, what didn't, and how to deploy it.
GPU capacity is globally distributed. You take what each region will give you. So you run several inference clusters on AWS, some on EKS and some on SageMaker HyperPod. Each one runs llm-d. Inside a cluster, llm-d's endpoint picker (EPP) does a good job. It sends each request to a good pod.
Across clusters, routing becomes more difficult. Requests often land in the wrong cluster, and then on a poor pod in that cluster. When you care about throughput above all, inter-token latency (ITL) matters more than time to first token (TTFT).
This post covers one day of building a fix. We stood up four EKS clusters in three regions and tried
several routing policies, first on a simulator and then on real GPUs. Some of our ideas failed, and the failures taught us the most. So we recount them here, for you. The result is a small llm-d plugin. At 160 concurrent streams on real vLLM it delivers 13% more tokens per second than the best upstream policy, with half the TTFT.
The first idea
The plan was simple. Put a global entrypoint cluster in front of everything, and run llm-d there too.
- Each inference cluster registers with the entrypoint when it starts and leaves when it stops.
- Each cluster's latency sidecar writes its best pod's ITL into Redis in the entrypoint region.
- A custom sidecar in the entrypoint reads Redis and returns the cluster with the lowest ITL.
- The global llm-d sends the request there, and that cluster's llm-d picks the pod.
Before building it, we read the llm-d and Gateway API Inference Extension docs and code. Three parts of the plan would not have worked as written.
- An InferencePool only selects Pods. It matches Pods by label in its own namespace. It can't point at a Service, an ExternalName or an IP outside the cluster. So "show up as an external service" was out.
- The latency predictor doesn't pick endpoints. The EPP sends the predictor sidecar a feature vector for each candidate pod, such as KV cache use, queue depth and input length. The sidecar sends back a predicted TTFT and TPOT. A scorer plugin inside the EPP does the ranking. The feature vector doesn't say which endpoint it describes, so a custom sidecar can't map a Redis value back to a cluster. The choice has to happen in a plugin, written in Go and compiled into the EPP.
- A proxy has no metrics. The EPP scrapes vLLM metrics from every endpoint. A plain proxy has none, so the EPP treats it as saturated.
We also suspected the signal itself. The best pod's ITL is one snapshot. It says nothing about spare
capacity. If every request chases the lowest number, they all go to one cluster. We kept the idea in
the test anyway. We wanted to see it fail.
llm-d already has a hub mode
llm-d-router v0.11.0 ships Alpha "hub" plugins. A hub EPP treats whole clusters as its endpoints.
-
multicluster-file-discoveryreads a list of clusters from a file and reloads it when the file changes.
-
multicluster-metrics-data-sourcescrapes each leaf cluster's EPP for pool averages, such as KV cache use and queue depth.
- Two scoring policies are shipped.
loadranks clusters on those pool averages.latencylearns each cluster's TTFT from the responses the hub sees.
Nothing in hub mode scores on ITL. The leaf EPP records an ITL histogram, but the hub's metrics extractor reads gauges and counters only. So for throughput routing, we had to write something.
Why not route on the TPOT llm-d already exports?
The leaf EPP exports time per output token (TPOT) as well as ITL. The two are close, but they differ.
-
llm_d_epp_request_streaming_tpot_secondsgives one number per request. It is (end-to-end time − TTFT) ÷ (output tokens − 1), the request's average decode speed.
-
llm_d_epp_request_streaming_itl_secondsrecords every gap between streamed chunks. It shows the spikes, such as a new request's prefill stalling the whole batch.
The averages come out about the same. The tails differ. A request can have a fine TPOT and still contain a few 150 ms stalls. We saw exactly that on a T4 later in this post.
So why doesn't llm-d route on a metric it exports? Because measured TPOT lags. It tells you what the last requests got, after the fact. Send traffic to whoever had the best recent number, and every request goes to the same place until that number gets worse. Our first idea failed this way.
llm-d predicts TPOT instead. Inside a cluster, predicted latency routing sends each pod's live load to a predictor. The load includes KV cache use, queue depth, running requests and prefix match. The predictor returns a TTFT and a TPOT, and llm-d scores on both. By default it weights TTFT 0.8 and TPOT 0.2. A request can set a TPOT target with the
x-llm-d-slo-tpot-ms header. Measured TPOT is the predictor's training data.That design assumes every pod in a pool is the same. The docs say "heterogeneous pools are not yet modeled".
With the same GPU and the same model, TPOT differences come mostly from batch size, and the load scorers already see batch size. TTFT varies more, because of queueing and prefix cache hits, so it gets the bigger weight.
Across clusters that assumption breaks. Different GPUs give different TPOT for reasons that load can't explain. Our plugin fills that gap the same way llm-d does inside a cluster. It predicts from load, and it doesn't react to past numbers. The prediction is a fitted ITL-versus-load line per cluster, used with the hub's live count of requests in flight.
There was one more gap. The hub's Envoy forwards to
ip:port only. AWS load balancers publish host names. The upstream guide says AWS is "not supported yet". We worked around it. An internal NLB keeps its private IPs for its whole life, so we resolve the name once and register the IP.The test bed
Cluster | Region | Role |
|---|---|---|
gr-hub | us-east-2 | Entrypoint. Hub EPP, registrar, load generator |
gr-leaf-use2 | us-east-2 | Leaf, 2.6 ms from the hub |
gr-leaf-use1 | us-east-1 | Leaf, 18 ms from the hub |
gr-leaf-euw2 | eu-west-2 | Leaf, 93 ms from the hub |
Each cluster has its own VPC with a unique /16 from one /14 block. A Transit Gateway in each region is peered with the one in us-east-2. TGW peering needs static routes on both sides, so adding a region is a script step. One route for the whole /14 means each VPC route table needs one entry, however many leaves join.
Each leaf runs the llm-d router with an internal NLB in front.
We started with the llm-d inference simulator in place of vLLM. It let us set each cluster's speed on
purpose. Idle ITL was 15 ms in us-east-2, 25 ms in us-east-1 and 40 ms in eu-west-2. ITL doubled at a full batch of 16. Later we swapped in real vLLM on GPUs.
The registry
We kept the Redis idea, as a registry. The store is ElastiCache for Valkey in the hub region, with TLS on.
Each leaf runs a small publisher. Every 2 seconds it reads its own leaf's metrics and writes one record
to
llmd:cluster:<name> with a 10 second expiry.- It reads the leaf EPP for ready pods, queue depth, and the streaming ITL histogram.
- It reads every model server pod for that pod's ITL and running requests.
- It fits a straight line, ITL = a + b × running requests per pod, over the last 300 per-pod samples.
- It resolves its own NLB name to an IP for the hub.
- If its cluster has been idle for 15 seconds, it sends one small request through its own router. An idle cluster has no fresh ITL, and a cluster with no ITL never gets picked.
On the hub, a registrar turns the live records into the hub's cluster file each second. When the
publishers started, the hub's list grew from one cluster to three with no hand edits.
The plugin
cluster-itl-scorer is about 250 lines of Go in a copy of llm-d-router v0.11.0. It polls Valkey every
500 ms. It plays two roles.- Filter. It drops a cluster whose record is missing or more than 10 seconds old. That takes under a second. Kubernetes can take up to a minute to push a changed ConfigMap into a pod, so the cluster file alone is slow.
- Scorer. It ranks clusters by the ITL a new request should expect. Version one had two modes.
best-podused the best pod's ITL, which was our original idea.predictedused the fitted line at the load the cluster would have after this request. It counts that load from the hub's own live count of requests in flight.
We built it in CodeBuild. Source went to S3,
go test ran in a golang container, and the image went to ECR.Round one on the simulator
At 32 streams the ITL policies won on every throughput number. They delivered 30% more tokens per second than upstream
load , and ITL p90 was 28 ms against 50 ms. load spread traffic evenly, including to the cluster that decodes 2.7 times slower.At 96 streams both ITL policies showed herd behavior. Every request saw the same "best" cluster and went there, until that cluster overloaded. Best-pod sent 85% of requests to us-east-2. Its 48 decode slots overflowed, TTFT p50 rose to 1.9 seconds, and throughput fell below
load . A fresh low ITL pulls traffic until the next 2 second publish shows the damage.predicted herded too, and it shouldn't have. We had listed the hub's in-flight count as an optional
input. The EPP's loader only creates a default data producer for required inputs. So the count was always zero, and the plugin was blind to its own decisions. A one-word fix and a test fixed it.The fixed
predicted mode was better, but TTFT was still high. The fitted line said the fast cluster stayed the cheapest well past its batch limit, so requests queued there. A queued request has a fine ITL once it runs. It waits first. So we added a weight for queueing. With queuePenalty: 10 , the plugin fills the nfastest cluster up to its batch limit and then moves to the next one. We call it spill.At 96 streams spill gave 2,010 tokens/s, the best of any policy, against 1,824 for
load and 1,900 for
latency . TTFT p50 fell from 888 ms to 320 ms.Breaking things on purpose
We killed a leaf under load to see what the hub does.
Clean shutdown. The publisher deletes its key, then the router stops. Zero errors. The filter dropped
the cluster within a second, and traffic moved.
Crash. The publisher and router die together, so nobody deletes the key. This caused a full outage for about 15 seconds. The two healthy clusters were idle. The dead cluster's load balancer reset each connection at once, so its requests failed in milliseconds. Its in-flight count fell to zero. With its last record still fresh, it looked like the best cluster, and the hub sent it everything. Any load-aware scorer will read a fast failure as spare capacity.
The fix was a health check from the hub. The plugin probes each cluster every second and drops it after two failures. The outage fell to about 3 seconds. Retries would hide those last seconds. The EPP protocol can return fallback endpoints, and Envoy can retry on connect failure. We didn't build that yet.
Registry outage. We rebooted the Valkey node under load. Zero errors, and throughput stayed flat. The plugin kept its last records, and when they went stale, its filter kept every cluster. A dead registry makes routing worse. It doesn't stop it.
The reboot also flushed the data. As publishers came back one by one, the registrar wrote a cluster file with one cluster in it for a second. Now the registrar keeps a cluster until its key has been gone for 15 seconds. This gave us a rule. Use two removal paths. The filter removes a cluster fast and can add it back fast. The file changes slowly and only needs to be right in the end.
Deploys looked like outages. Restarting a publisher dropped its cluster for 2 seconds. The old pod's
exit handler deleted the key after the new pod had already written it. Now each record names its writer.
On exit, a small Lua script deletes the key only if this pod still owns it. A readiness probe also waits
for the new pod's first publish before Kubernetes stops the old one.
Real GPUs
We asked for three GPU types in three regions and got what was free.
Leaf | Asked for | Got |
|---|---|---|
us-east-2 | g6e.xlarge (L40S) | g6.xlarge (L4) |
us-east-1 | g5.xlarge (A10G) | g5.xlarge (A10G) |
eu-west-2 | g6.xlarge (L4) | g4dn.xlarge (T4), on the third try |
Two regions returned
InsufficientInstanceCapacity in every zone for 20 minutes. Node groups with a list of instance types got us through. This is the same capacity problem that makes multi-cluster routing worth doing. Our home region also ended up with a slower GPU than us-east-1, so a fixed region preference would have been wrong.We ran Qwen2.5-1.5B-Instruct on vLLM v0.19.1, one GPU per pod, 2 pods per leaf, 32 sequences each. The leaf router and publisher needed no changes. They read real vLLM metrics the same way.
Real vLLM did not look like the simulator. For a small model, ITL barely rises with batch size. On the A10G it stayed near 13 ms from 4 to 32 running requests. What moves ITL is spikes. When a new request's prefill runs, every decode in the batch waits a step. One 2 second window on the T4 averaged 160 ms. Our straight line chased the spikes and came out wrong. Dropping samples above three times the median fixed the fit. The order of the clusters stayed clear the whole time. us-east-1 was fastest, then us-east-2, then eu-west-2.
The table shows results at 160 streams, which is 85% of capacity.
Policy | tokens/s | TTFT p50 / p90 (ms) | ITL p90 (ms) |
|---|---|---|---|
upstream load | 6,540 | 541 / 4,119 | 25.9 |
upstream latency (TTFT) | 6,813 (mean of 3) | 423 / 3,306 | 22.0 |
best-pod ITL (the first idea) | 4,575 | 4,381 / 7,288 | 14.6 |
ITL spill | 7,717 (mean of 3) | 233 / 1,569 | 20.8 |
load and best-pod are single runs. Spill won on all three of its runs against latency . The gain was 13% in tokens per second, with about half the TTFT.Best-pod herded worse on GPUs than on the simulator. It sent 94% of requests to us-east-1. Its ITL figure looks great, because the requests that do run are on the fastest GPU. Its throughput was the worst of the four. ITL alone is the wrong goal. You have to count the wait.
Spill and upstream
latency sent almost the same share of traffic to each region over a run. The gain came from timing. Spill moves the next request at the moment the fast cluster fills. latency learns from TTFT it has already seen, so it reacts after the queue has formed.At light load, upstream
latency had the best TTFT. ITL-aware routing still gave 5% more tokens per second.How to deploy it
The plugin ships inside your own EPP image. The rest is the standard llm-d router Helm chart with a
different image and config. Here are the steps.
1. Add the plugin to llm-d-router. Clone the release your chart uses. We used v0.11.0.
git clone --depth 1 --branch v0.11.0 https://github.com/llm-d/llm-d-router.gitCopy the plugin into
pkg/epp/framework/plugins/scheduling/scorer/clusteritl/ . Then register it in
cmd/epp/runner/runner.go . These two lines are the only upstream change.import "github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/scheduling/scorer/clusteritl"
fwkplugin.Register(clusteritl.PluginType, fwkplugin.StabilityAlpha, clusteritl.Factory)The Redis client and the test Redis are already in
go.mod , so the dependencies don't change.2. Build and push the image. Use the repo's own
Dockerfile.epp .docker run --rm -v $PWD:/w -w /w golang:1.26.8 \ go test ./pkg/epp/framework/plugins/scheduling/scorer/clusteritl/...docker build -f Dockerfile.epp --build-arg TARGETOS=linux --build-arg TARGETARCH=amd64 \ -t <account>.dkr.ecr.us-east-2.amazonaws.com/llm-d-epp:clusteritl .docker push <account>.dkr.ecr.us-east-2.amazonaws.com/llm-d-epp:clusteritlCodeBuild ran these same steps. EKS nodes pull from ECR in the same account with no extra setup.
3. Set up the leaves and the registry. Each leaf runs the llm-d router with metrics auth turned off, so the hub can read them. An internal NLB exposes port 80 for traffic and port 9090 for metrics. The publisher runs beside the router. On the hub, the registrar keeps the
mc-hub-clusters ConfigMap current. Each entry looks like this.endpoints: - name: leaf-use1 # must match the publisher's cluster name address: "10.202.123.11" # an IP, since the hub's Envoy can't use a host name port: "80" labels: {metricsAddress: "10.202.123.11", metricsPort: "9090"}4. Install the hub. Layer two values files on the chart.
helm upgrade --install hub oci://ghcr.io/llm-d/charts/llm-d-router-standalone --version v0.11.0 \ -n llm-d -f hub.values.yaml -f hub-itl-spill.values.yamlhub.values.yaml comes from the upstream multi-cluster guide. It turns on experimental plugins and mounts the cluster list. hub-itl-spill.values.yaml swaps in the image and the plugin config.router: epp: image: registry: <account>.dkr.ecr.us-east-2.amazonaws.com repository: llm-d-epp tag: clusteritl pluginsConfigFile: "itl.yaml" pluginsCustomConfig: itl.yaml: | apiVersion: llm-d.ai/v1alpha1 kind: EndpointPickerConfig plugins: - type: multicluster-file-discovery name: discovery parameters: {path: /etc/epp/clusters/clusters.yaml, watchFile: true} - type: cluster-itl-scorer name: itl parameters: redisAddr: <valkey-endpoint>:6379 redisTLS: true mode: predicted queuePenalty: 10 # spill: fill the fastest cluster, then move on staleAfter: 10s healthInterval: 1s # hub-side liveness probe healthFailures: 2 - type: max-score-picker name: picker - type: single-profile-handler name: profile-handler schedulingProfiles: - name: default plugins: - pluginRef: itl weight: 1 # one weighted entry makes it a filter and a scorer - pluginRef: picker dataLayer: discovery: endpoints: {pluginRef: discovery}5. Check it works. The hub EPP log should show
auto-created default producer for
inflight-load-producer , with itl as the consumer. That proves the plugin can see its own decisions. The parsed config should list itl/cluster-itl-scorer under Filters . Run the EPP at verbosity 4 to log the cost it computes for each cluster.Three things caused us problems.
- Match the config version to the image. The v0.11.0 image only accepts
llm-d.ai/v1alpha1. Guides on the main branch usellm-d.ai/v1.
- Mark the in-flight count as required. If your plugin lists it as optional, the EPP never creates the producer, and the count stays at zero.
- Turn on experimental plugins. Hub plugins and ours are Alpha. Without
--allow-experimental-plugins, the EPP refuses to start.
Plugin settings are read once, at startup. A change to
queuePenalty or mode needs a helm upgrade .What we'd tell you to do
- Use llm-d hub mode for the global layer. Proxies and fake services aren't needed. Peer clusters are endpoints in a file, and the hub reads their routers' metrics.
- Route on expected ITL, and treat each cluster's batch limit as a soft wall. Fill the fastest cluster, then spill. Never queue on a fast cluster while a slower one has free decode slots.
- Use the hub's own in-flight count. Leaf metrics are seconds old. The hub knows what it sent a millisecond ago. Mark it required in your plugin.
- Keep Redis, as a registry with a short expiry. Add owner checks on delete and a grace period on removal.
- Check liveness from the hub. A dead cluster looks idle. Don't wait for a key to expire.
- Fit the ITL model robustly. Real ITL is spiky. A two-number line per cluster is enough if you drop the spikes.
- Plan CIDRs and quotas per region on day one.
What's left
- Retries with fallback endpoints, to hide the last seconds of a crash.
- TLS between hub and leaves. Today hub mode speaks plain HTTP over the private network.
- A HyperPod leaf. HyperPod's Inference Gateway already runs an llm-d EPP, so it should join the same way.
- A bigger model, where batch size moves ITL more and the fitted line matters more.
- Upstream. The plugin is small, and an ITL scorer for hub mode would help anyone serving throughput-bound traffic.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article