💀 Your LLM Is Not Slow—Your Architecture Is
Most of your LLM’s wait isn’t the model. Trace one request, fix the architecture around it, and cut latency 39% without touching the model.
You ask your LLM app a question. 6.2 seconds later, it answers. So you blame the model.But the two model calls took 2.7 seconds. The other 3.5 seconds came from everything around them. 💀
Before you swap in a "faster" model, look at where the time actually went.
One request, traced
One illustrative latency budget (not a benchmark) showing where the time can go. Tools ran one after another, the prompt was bloated, and nothing streamed. The model's two calls: 2.7 s. The three tool calls alone: another 2.7 s.
Myth: "RAG is slow because of the vector DB"
Published RAG latency guides put embedding and vector search at tens of milliseconds. Reranking is the heavy retrieval stage, and it scales with how many candidates you feed it. The bigger bill comes after retrieval: time to first token (TTFT) grows with prompt length. In one published benchmark, 32K input tokens meant a 472 ms TTFT and 123K tokens meant about 2.2 s. That's 3.75× the tokens for 4.7× the wait.
So: retrieve → filter → rerank fewer → compress → generate.
Fix 1: Stop making independent calls wait in line
Three tools at 900 ms each: sequential is 2.7 s, parallel is 0.9 s. Latency follows the slowest call, not the number of calls.
Published results top out at 3.7× faster, on tasks with independent calls. If call B needs A's output, it can't run in parallel.
Fix 2: Send less, cache the rest
Input length drives TTFT. Output length drives total time. On Amazon Bedrock, prompt caching reports up to 85% lower latency and 90% lower cost for supported models. It helps most on long prompts and barely matters on short ones. Cache the static prefix (system prompt, tool definitions, reference docs), plus embeddings, retrieval results, and tool results you keep recomputing.
Fix 3: Stream the answer
TTFT ≠total latency. Streaming doesn't make generation faster. It lets the user see it sooner.
The scoreboard
| Change | Effect |
|---|---|
| Run the 3 tools in parallel | −1,800 ms |
| Rerank 20 candidates, not 50 | −240 ms |
| Trim and cache the prompt (prefill 800 → 400 ms) | −400 ms |
| Total latency | 6.21 s → 3.77 s (−39%) |
| Stream the answer | first token at 2.57 s (−59% wait) |
Same model. Same task. Not one line of the model changed.
Measure before you optimize
Averages hide the pain. In an AWS test of the GPT-5.6 models in July 2026, worst-case TTFT ran 2.1–2.5× the median on Bedrock (4.6–6.6× on the OpenAI API). It was a single-region, point-in-time snapshot, but the lesson holds: track p50, p95, and p99 for every stage.
- [ ] Trace every stage, not just the model call
- [ ] Parallelize independent calls
- [ ] Cut candidates and context before generation
- [ ] Cache prefixes, embeddings, retrievals, tool results
- [ ] Stream, and track TTFT and total latency separately
- [ ] Set a latency budget per stage
Before replacing your model, trace your request. Your model might not be the bottleneck. Your architecture might be.
The trace is illustrative: stage timings were chosen inside published typical ranges, not measured on one system, and the savings are arithmetic on that trace (the 800 → 400 ms prefill cut is a conservative assumption). Figures from the sources below are as reported by them.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article