How to Build an LLM Inference Benchmark That Doesn't Lie
Why most LLM benchmarks overstate performance, and how to design one that reflects what your users will actually experience in production.
This post is for ML platform engineers running self-hosted vLLM on GPU instances who need benchmark results they can commit to in an SLA conversation. If you're using a managed inference API, some of this applies — but the server-side metrics section won't.
The Problem
You benchmark your LLM inference setup. The numbers look great — sub-400ms P50 at 8K input tokens. You ship to production. Users report 600ms+ latency. What happened?
The benchmark lied. Not intentionally — but through a series of small methodological choices that each tilted the results in your favour. Synthetic warmups, tiny prompt pools, client-side timing that hides network jitter, and cache hit rates that don't match real traffic patterns.
I spent the past few weeks benchmarking models on various EC2 GPU instances with vLLM. My first benchmark script reported P50 TTFT of ~380ms. By the third iteration — same hardware, same model, same config — it reported ~640ms. Nothing about the infrastructure changed. The benchmark got more honest. This post explains exactly what changed between v1 and v3, and why the worse number is the one you want.
Three Ways Benchmarks Lie
1. Explicit Warmup Inflates Cache Hit Rates
Most benchmark scripts include a warmup phase:
# The lie: explicit warmup before measurementprint("Warming prefix cache (5 requests)...")for _ in range(5): send_request(cache_hit=True)
# NOW start measuringresults = run_benchmark()
This guarantees 95%+ prefix cache hit rates from the first measured request. In production, your cache starts cold. The first N requests for each unique prompt prefix pay the full prefill cost. If your system serves 50 different bot configurations, the cache needs time to populate — and under high concurrency, cache pressure causes evictions.
The fix: Don't warmup. Let the cache build organically from your request stream, exactly as it would in production.
2. Tiny Query Pools Create Artificial Cache Hits
# The lie: 3 unique queries, repeated foreverCACHED_QUERIES = [ "Customer: I need help with billing...", "Customer: I want to cancel...", "Customer: My order hasn't arrived...",]query = random.choice(CACHED_QUERIES) # Only 3 options!
With 3 unique queries and a 5-request warmup, you're guaranteed a cache hit on every measured request — the KV cache has already computed all 3 prefixes. In production, you might have hundreds or thousands of unique conversation transcripts, each with a shared system prompt prefix but unique suffixes.
The fix: Use 50-100+ unique queries. The system prompt prefix will still cache (correctly), but the unique suffixes won't — giving you realistic cache hit rates.
3. Client-Side Timing Includes Network Latency You Can't Control
start = time.perf_counter()response = client.responses.create(...) # Includes network round-tripelapsed_ms = (time.perf_counter() - start) * 1000
This measures end-to-end from your laptop, which includes:
- TCP connection setup
- TLS handshake (if HTTPS)
- Network latency to the instance (variable, 20-80ms depending on location)
- Actual inference time on the GPU
When you compare your numbers to a competitor's, you might be measuring different things. Their benchmark might run from a machine in the same AZ as the inference server. Yours might cross the internet.
The fix: Report both client-side and server-side metrics. vLLM exposes a
/metrics Prometheus endpoint with server-measured latencies:Also: run your benchmark client from within the same VPC, same AZ, and same placement group as your inference instance. This eliminates cross-AZ and cross-internet latency entirely from your client-side numbers, giving you a clean measure of application-layer overhead rather than network topology.
# Scrape vLLM metrics before and after your benchmark runmetrics_before = requests.get(f"{ENDPOINT}/metrics").textrun_benchmark()metrics_after = requests.get(f"{ENDPOINT}/metrics").text
# Server-side metrics (no network noise):# - vllm:e2e_request_latency_seconds# - vllm:time_to_first_token_seconds# - vllm:request_prefill_time_seconds# - vllm:request_decode_time_seconds# - vllm:prefix_cache_hits_total / vllm:prefix_cache_queries_total# - vllm:time_per_output_token_seconds (inter-token latency)
The server metrics tell you the truth. The client metrics tell you what users experience.
Modeling Real Workloads
The most impactful decision in your benchmark is how you model the prompt structure. Get this wrong and nothing else matters.
Understand Your Prompt Anatomy
Most production LLM workloads have a layered prompt structure:
┌─────────────────────────────────────────────┐│ Static Prefix (~80-90% of input tokens) │ ← Cacheable│ System instructions, schemas, few-shot ││ examples, compliance rules │├─────────────────────────────────────────────┤│ Dynamic Middle (~5-10%) │ ← NOT cacheable│ Bot config, user settings, session state │├─────────────────────────────────────────────┤│ Dynamic Tail (~5-10%) │ ← NOT cacheable│ Unique conversation / user input │└─────────────────────────────────────────────┘
Most guides only distinguish "static" from "dynamic" — but the middle layer is where multi-tenant systems get surprised. Each unique bot configuration creates a different cache prefix from that point downward, so 10 bot configs means 10 distinct cacheable prefixes rather than one. Your benchmark needs to model that diversity.
Prefix caching (in vLLM, TensorRT-LLM, etc.) matches the longest common prefix across requests. The cache hits the static prefix, then misses the moment the dynamic content diverges. This means:
- If 30% of your tokens are the static prefix → ~30% cache hit rate (theoretical max)
- But only if the static prefix is truly identical across requests — vLLM uses SHA256-based block hashing, so only full KV cache blocks are eligible for reuse. A prefix that falls 1 token short of a full block boundary won't cache. In practice, keep your static prefix length a clean multiple of vLLM's block size (default 16 tokens) to avoid leaving cache hits on the table.
The Subtlety: "Static" Isn't Always Static
Many system prompts look static but contain dynamic interpolations:
You are a support agent for {{company_name}}.Current date: {{today}}Customer tier: {{tier}}Allowed tools: {{tool_list}}
Each unique combination of template variables creates a different prefix from the cache's perspective. If you have 10 bot configurations, you have 10 different cacheable prefixes — not one. Your benchmark needs to model this:
# Realistic: 50 unique dynamic configs (models multiple bots)def build_prompt(i): static_prefix = SHARED_TEMPLATE # Same for all bots (~7K tokens) dynamic_config = generate_bot_config(i) # Varies per bot (~500 tokens) user_input = generate_conversation(i) # Unique per request (~500 tokens) return static_prefix + dynamic_config + user_input
Calculating Your Real Cache Rate
The formula:
effective_cache_rate = (static_prefix_tokens) / (total_input_tokens)
But that's the theoretical maximum. In practice, cache pressure at high concurrency can evict prefixes. Your measured cache rate (from
/metrics ) tells you the actual number. If your measured rate is close to your theoretical max, you're caching efficiently. If it falls significantly short, you're either under cache pressure or your prompts have more dynamic content than you think.What to Report (And What to Sanity-Check)
Report These
Metric | Source | What it tells you |
|---|---|---|
Client P50/P95/P99 | Your benchmark script | What end users experience (includes network) — P99 is what matters in SLA conversations |
Server avg E2E | vLLM /metrics | True inference time (no network) |
Server TTFT | vLLM /metrics | Time to first token — dominated by prefill |
Inter-token latency (ITL) | vLLM /metrics ( vllm:time_per_output_token_seconds ) | What users feel after the first token in streaming — the per-token rhythm. Matters more than TTFT for chatbot UX |
Server decode time | vLLM /metrics | Decode phase — dominated by output token count |
Prefix cache hit % | vLLM /metrics | Whether your prompt structure is actually caching |
Avg input tokens | API response usage | Confirms you're hitting your target input length |
Request count | Your script | Statistical significance — 20 requests is not enough at high concurrency |
Finding the Saturation Point
Your concurrency sweep will show latency climbing as concurrency increases while throughput continues to grow. The saturation point is where P95 latency crosses your SLA threshold while tokens/second is still increasing — that's your production concurrency ceiling. In practice: plot P95 latency and throughput on the same axis across your concurrency levels. The crossing point is what you commit to ops.
Sanity Checks
Problem to check | Symptom | Likely diagnosis |
|---|---|---|
Network overhead | Client P50 − Server avg E2E > 100ms, or gap shifts between runs | Client is far from server, or network is noisy — run more requests or move client into the same AZ |
Cache not behaving as designed | Measured hit rate significantly below theoretical max | Dynamic interpolations in your "static" prefix, or too many unique prefixes causing KV cache eviction |
Decode anomaly | Decode time doesn't scale linearly with output tokens, or doubles between identical runs | Batching contention, memory pressure, or chunked prefill config mismatch |
Output length masking tail latency | P95 >> P50 | Fixed max_output_tokens ceiling hiding variable output length distribution — sample from a realistic range instead |
Noisy results | P50 varies >±5% across identical runs | Too few requests per concurrency level, background GPU processes, or network instability |
One config flag that changes everything:
--enable-chunked-prefill . If you're using it (it's on by default in recent vLLM builds), be aware that it splits long prefills across multiple scheduler steps to reduce TTFT at the cost of higher ITL. This means TTFT looks better and inter-token latency looks worse compared to a run without it. If you're comparing across configs, make sure chunked prefill is consistently on or off — mixing runs will produce incomparable numbers.A Benchmark Structure That Works
If you prefer a pre-built harness, vLLM officially recommends GuideLLM for production server benchmarking and
inference-perf for EKS/Kubernetes environments — both handle dataset management, workload patterns, and report generation. NVIDIA GenAI-Perf (bundled with Triton) and the vllm bench CLI are also solid options. All of these have warmup behaviours enabled by default, so review their defaults against the principles above before trusting their numbers.After three iterations, here's the structure I landed on:
BENCHMARK_VERSION = "v3.1" # Track this!
# Configuration that matches productionTOTAL_INPUT_TOKENS = 1000CACHE_RATE = 0.30 # Adjust to match your actual workload — see "Prompt Anatomy" section aboveREQUESTS_PER_LEVEL = 40 # Minimum to observe queue behavior at each concurrency level; # increase to 100+ if your P95 variance between runs is >5%CONCURRENCY_LEVELS = [1, 2, 4, 8, 10] # 10 = production peak concurrency for this workloadMAX_OUTPUT_TOKENS = 250
# NO warmup — cache builds organicallydef run_sweep(concurrency, num_requests=40): # Build request mix: CACHE_RATE cached prefix + remainder fully unique hits = int(num_requests * CACHE_RATE) request_types = [True] * hits + [False] * (num_requests - hits) random.shuffle(request_types)
# Fire all at once, collect results with ThreadPoolExecutor(max_workers=concurrency) as executor: futures = [executor.submit(send_request, t) for t in request_types] results = [f.result() for f in as_completed(futures)] return results
# Collect BOTH client and server metricsfor concurrency in CONCURRENCY_LEVELS: server_before = scrape_vllm_metrics() results = run_sweep(concurrency) server_after = scrape_vllm_metrics()
# Report both perspectives client_p50 = percentile(results, 50) server_avg = compute_delta(server_before, server_after)
Key design decisions:
- No warmup — organic cache building
- 50+ unique queries — realistic diversity
- 40 requests per level — statistical significance (increase to 100+ if P95 is noisy)
- Both client and server metrics — full picture
- Shuffled request order — prevents artificial batching patterns
- Version everything — track benchmark script version, warmup on/off, unique query count, static/dynamic token split, vLLM version and flags, and whether structured output was enabled in every results file. When you compare Tuesday's run to Thursday's, you need to know whether the benchmark changed, not just the model config.
In production on AWS, forward vLLM's Prometheus endpoint to CloudWatch using the AWS Distro for OpenTelemetry (ADOT) collector. Add this scrape config to your ADOT
config.yaml :receivers: prometheus: config: scrape_configs: - job_name: vllm scrape_interval: 15s static_configs: - targets: ["localhost:8000"] # vLLM metrics endpoint
exporters: awsemf: namespace: vLLM/Inference region: us-east-1 log_group_name: /vllm/metrics dimension_rollup_option: NoDimensionRollup
service: pipelines: metrics: receivers: [prometheus] exporters: [awsemf]This publishes vLLM metrics as CloudWatch EMF (Embedded Metric Format) under the
vLLM/Inference namespace, where you can build alarms on TTFT, cache hit rate, and queue depth without running a separate Grafana stack.One Thing to Change Right Now
If you take nothing else from this post: open your benchmark script and remove the warmup phase. That single change — more than query pool size, more than metric collection — is what most inflates numbers. Your P50 will go up. Your cache hit rate will drop. Both of those are correct. Ship the honest number, not the optimistic one.
If you want to go further: add the vLLM
/metrics scrape, expand your query pool to 50+, and add a BENCHMARK_VERSION constant to every results file. In that order. The honest number is the useful one.Resources
- NVIDIA GenAI-Perf — pre-built benchmarking harness (note: has its own warmup defaults to account for)
- vllm bench CLI — built into recent vLLM releases; useful for quick sweeps directly on the instance
- CloudWatch Agent Prometheus scrape config — for forwarding vLLM
/metricsto CloudWatch in production
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article