AWS Builder Center

How to Build an LLM Inference Benchmark That Doesn't Lie

Why most LLM benchmarks overstate performance, and how to design one that reflects what your users will actually experience in production.

This post is for ML platform engineers running self-hosted vLLM on GPU instances who need benchmark results they can commit to in an SLA conversation. If you're using a managed inference API, some of this applies — but the server-side metrics section won't.

The Problem

You benchmark your LLM inference setup. The numbers look great — sub-400ms P50 at 8K input tokens. You ship to production. Users report 600ms+ latency. What happened?
The benchmark lied. Not intentionally — but through a series of small methodological choices that each tilted the results in your favour. Synthetic warmups, tiny prompt pools, client-side timing that hides network jitter, and cache hit rates that don't match real traffic patterns.
I spent the past few weeks benchmarking models on various EC2 GPU instances with vLLM. My first benchmark script reported P50 TTFT of ~380ms. By the third iteration — same hardware, same model, same config — it reported ~640ms. Nothing about the infrastructure changed. The benchmark got more honest. This post explains exactly what changed between v1 and v3, and why the worse number is the one you want.

Three Ways Benchmarks Lie

1. Explicit Warmup Inflates Cache Hit Rates

Most benchmark scripts include a warmup phase:
# The lie: explicit warmup before measurement
print("Warming prefix cache (5 requests)...")
for _ in range(5):
send_request(cache_hit=True)

# NOW start measuring
results = run_benchmark()

This guarantees 95%+ prefix cache hit rates from the first measured request. In production, your cache starts cold. The first N requests for each unique prompt prefix pay the full prefill cost. If your system serves 50 different bot configurations, the cache needs time to populate — and under high concurrency, cache pressure causes evictions.
The fix: Don't warmup. Let the cache build organically from your request stream, exactly as it would in production.

2. Tiny Query Pools Create Artificial Cache Hits

# The lie: 3 unique queries, repeated forever
CACHED_QUERIES = [
"Customer: I need help with billing...",
"Customer: I want to cancel...",
"Customer: My order hasn't arrived...",
]
query = random.choice(CACHED_QUERIES) # Only 3 options!

With 3 unique queries and a 5-request warmup, you're guaranteed a cache hit on every measured request — the KV cache has already computed all 3 prefixes. In production, you might have hundreds or thousands of unique conversation transcripts, each with a shared system prompt prefix but unique suffixes.
The fix: Use 50-100+ unique queries. The system prompt prefix will still cache (correctly), but the unique suffixes won't — giving you realistic cache hit rates.

3. Client-Side Timing Includes Network Latency You Can't Control

start = time.perf_counter()
response = client.responses.create(...) # Includes network round-trip
elapsed_ms = (time.perf_counter() - start) * 1000

This measures end-to-end from your laptop, which includes:
  • TCP connection setup
  • TLS handshake (if HTTPS)
  • Network latency to the instance (variable, 20-80ms depending on location)
  • Actual inference time on the GPU
When you compare your numbers to a competitor's, you might be measuring different things. Their benchmark might run from a machine in the same AZ as the inference server. Yours might cross the internet.
The fix: Report both client-side and server-side metrics. vLLM exposes a  /metrics  Prometheus endpoint with server-measured latencies:
Also: run your benchmark client from within the same VPC, same AZ, and same placement group as your inference instance. This eliminates cross-AZ and cross-internet latency entirely from your client-side numbers, giving you a clean measure of application-layer overhead rather than network topology.
# Scrape vLLM metrics before and after your benchmark run
metrics_before = requests.get(f"{ENDPOINT}/metrics").text
run_benchmark()
metrics_after = requests.get(f"{ENDPOINT}/metrics").text

# Server-side metrics (no network noise):
# - vllm:e2e_request_latency_seconds
# - vllm:time_to_first_token_seconds
# - vllm:request_prefill_time_seconds
# - vllm:request_decode_time_seconds
# - vllm:prefix_cache_hits_total / vllm:prefix_cache_queries_total
# - vllm:time_per_output_token_seconds (inter-token latency)

The server metrics tell you the truth. The client metrics tell you what users experience.

Modeling Real Workloads

The most impactful decision in your benchmark is how you model the prompt structure. Get this wrong and nothing else matters.

Understand Your Prompt Anatomy

Most production LLM workloads have a layered prompt structure:
┌─────────────────────────────────────────────┐
│ Static Prefix (~80-90% of input tokens) │ ← Cacheable
│ System instructions, schemas, few-shot │
│ examples, compliance rules │
├─────────────────────────────────────────────┤
│ Dynamic Middle (~5-10%) │ ← NOT cacheable
│ Bot config, user settings, session state │
├─────────────────────────────────────────────┤
│ Dynamic Tail (~5-10%) │ ← NOT cacheable
│ Unique conversation / user input │
└─────────────────────────────────────────────┘

Most guides only distinguish "static" from "dynamic" — but the middle layer is where multi-tenant systems get surprised. Each unique bot configuration creates a different cache prefix from that point downward, so 10 bot configs means 10 distinct cacheable prefixes rather than one. Your benchmark needs to model that diversity.
Prefix caching (in vLLM, TensorRT-LLM, etc.) matches the longest common prefix across requests. The cache hits the static prefix, then misses the moment the dynamic content diverges. This means:
  • If 30% of your tokens are the static prefix → ~30% cache hit rate (theoretical max)
  • But only if the static prefix is truly identical across requests — vLLM uses SHA256-based block hashing, so only full KV cache blocks are eligible for reuse. A prefix that falls 1 token short of a full block boundary won't cache. In practice, keep your static prefix length a clean multiple of vLLM's block size (default 16 tokens) to avoid leaving cache hits on the table.

The Subtlety: "Static" Isn't Always Static

Many system prompts look static but contain dynamic interpolations:
You are a support agent for {{company_name}}.
Current date: {{today}}
Customer tier: {{tier}}
Allowed tools: {{tool_list}}

Each unique combination of template variables creates a different prefix from the cache's perspective. If you have 10 bot configurations, you have 10 different cacheable prefixes — not one. Your benchmark needs to model this:
# Realistic: 50 unique dynamic configs (models multiple bots)
def build_prompt(i):
static_prefix = SHARED_TEMPLATE # Same for all bots (~7K tokens)
dynamic_config = generate_bot_config(i) # Varies per bot (~500 tokens)
user_input = generate_conversation(i) # Unique per request (~500 tokens)
return static_prefix + dynamic_config + user_input


Calculating Your Real Cache Rate

The formula:
effective_cache_rate = (static_prefix_tokens) / (total_input_tokens)

But that's the theoretical maximum. In practice, cache pressure at high concurrency can evict prefixes. Your measured cache rate (from  /metrics ) tells you the actual number. If your measured rate is close to your theoretical max, you're caching efficiently. If it falls significantly short, you're either under cache pressure or your prompts have more dynamic content than you think.

What to Report (And What to Sanity-Check)

Report These

Metric
Source
What it tells you
Client P50/P95/P99
Your benchmark script
What end users experience (includes network) — P99 is what matters in SLA conversations
Server avg E2E
vLLM  /metrics 
True inference time (no network)
Server TTFT
vLLM  /metrics 
Time to first token — dominated by prefill
Inter-token latency (ITL)
vLLM  /metrics  ( vllm:time_per_output_token_seconds )
What users feel after the first token in streaming — the per-token rhythm. Matters more than TTFT for chatbot UX
Server decode time
vLLM  /metrics 
Decode phase — dominated by output token count
Prefix cache hit %
vLLM  /metrics 
Whether your prompt structure is actually caching
Avg input tokens
API response  usage 
Confirms you're hitting your target input length
Request count
Your script
Statistical significance — 20 requests is not enough at high concurrency

Finding the Saturation Point

Your concurrency sweep will show latency climbing as concurrency increases while throughput continues to grow. The saturation point is where P95 latency crosses your SLA threshold while tokens/second is still increasing — that's your production concurrency ceiling. In practice: plot P95 latency and throughput on the same axis across your concurrency levels. The crossing point is what you commit to ops.

Sanity Checks

Problem to check
Symptom
Likely diagnosis
Network overhead
Client P50 − Server avg E2E > 100ms, or gap shifts between runs
Client is far from server, or network is noisy — run more requests or move client into the same AZ
Cache not behaving as designed
Measured hit rate significantly below theoretical max
Dynamic interpolations in your "static" prefix, or too many unique prefixes causing KV cache eviction
Decode anomaly
Decode time doesn't scale linearly with output tokens, or doubles between identical runs
Batching contention, memory pressure, or chunked prefill config mismatch
Output length masking tail latency
P95 >> P50
Fixed  max_output_tokens  ceiling hiding variable output length distribution — sample from a realistic range instead
Noisy results
P50 varies >±5% across identical runs
Too few requests per concurrency level, background GPU processes, or network instability
One config flag that changes everything:  --enable-chunked-prefill . If you're using it (it's on by default in recent vLLM builds), be aware that it splits long prefills across multiple scheduler steps to reduce TTFT at the cost of higher ITL. This means TTFT looks better and inter-token latency looks worse compared to a run without it. If you're comparing across configs, make sure chunked prefill is consistently on or off — mixing runs will produce incomparable numbers.

A Benchmark Structure That Works

If you prefer a pre-built harness, vLLM officially recommends  GuideLLM  for production server benchmarking and  inference-perf  for EKS/Kubernetes environments — both handle dataset management, workload patterns, and report generation. NVIDIA GenAI-Perf (bundled with Triton) and the  vllm bench  CLI are also solid options. All of these have warmup behaviours enabled by default, so review their defaults against the principles above before trusting their numbers.
After three iterations, here's the structure I landed on:
BENCHMARK_VERSION = "v3.1" # Track this!

# Configuration that matches production
TOTAL_INPUT_TOKENS = 1000
CACHE_RATE = 0.30 # Adjust to match your actual workload — see "Prompt Anatomy" section above
REQUESTS_PER_LEVEL = 40 # Minimum to observe queue behavior at each concurrency level;
# increase to 100+ if your P95 variance between runs is >5%
CONCURRENCY_LEVELS = [1, 2, 4, 8, 10] # 10 = production peak concurrency for this workload
MAX_OUTPUT_TOKENS = 250

# NO warmup — cache builds organically
def run_sweep(concurrency, num_requests=40):
# Build request mix: CACHE_RATE cached prefix + remainder fully unique
hits = int(num_requests * CACHE_RATE)
request_types = [True] * hits + [False] * (num_requests - hits)
random.shuffle(request_types)

# Fire all at once, collect results
with ThreadPoolExecutor(max_workers=concurrency) as executor:
futures = [executor.submit(send_request, t) for t in request_types]
results = [f.result() for f in as_completed(futures)]
return results

# Collect BOTH client and server metrics
for concurrency in CONCURRENCY_LEVELS:
server_before = scrape_vllm_metrics()
results = run_sweep(concurrency)
server_after = scrape_vllm_metrics()

# Report both perspectives
client_p50 = percentile(results, 50)
server_avg = compute_delta(server_before, server_after)

Key design decisions:
  • No warmup — organic cache building
  • 50+ unique queries — realistic diversity
  • 40 requests per level — statistical significance (increase to 100+ if P95 is noisy)
  • Both client and server metrics — full picture
  • Shuffled request order — prevents artificial batching patterns
  • Version everything — track benchmark script version, warmup on/off, unique query count, static/dynamic token split, vLLM version and flags, and whether structured output was enabled in every results file. When you compare Tuesday's run to Thursday's, you need to know whether the benchmark changed, not just the model config.
In production on AWS, forward vLLM's Prometheus endpoint to CloudWatch using the AWS Distro for OpenTelemetry (ADOT) collector. Add this scrape config to your ADOT  config.yaml :
receivers:
prometheus:
config:
scrape_configs:
- job_name: vllm
scrape_interval: 15s
static_configs:
- targets: ["localhost:8000"] # vLLM metrics endpoint

exporters:
awsemf:
namespace: vLLM/Inference
region: us-east-1
log_group_name: /vllm/metrics
dimension_rollup_option: NoDimensionRollup

service:
pipelines:
metrics:
receivers: [prometheus]
exporters: [awsemf]
This publishes vLLM metrics as CloudWatch EMF (Embedded Metric Format) under the  vLLM/Inference  namespace, where you can build alarms on TTFT, cache hit rate, and queue depth without running a separate Grafana stack.

One Thing to Change Right Now

If you take nothing else from this post: open your benchmark script and remove the warmup phase. That single change — more than query pool size, more than metric collection — is what most inflates numbers. Your P50 will go up. Your cache hit rate will drop. Both of those are correct. Ship the honest number, not the optimistic one.
If you want to go further: add the vLLM  /metrics  scrape, expand your query pool to 50+, and add a  BENCHMARK_VERSION  constant to every results file. In that order. The honest number is the useful one.

Resources

  •  NVIDIA GenAI-Perf  — pre-built benchmarking harness (note: has its own warmup defaults to account for)
  •  vllm bench CLI  — built into recent vLLM releases; useful for quick sweeps directly on the instance
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article