If you’re running LLM-powered applications in production, you know the pattern: every token processed-in prompts, context windows, or generated outputs-translates directly to API costs, inference latency and capacity issues. With models now supporting 128K+ token windows, the temptation to throw more context at every problem creates a compounding cost spiral.

After reviewing over hundereds of solutions (from frameworks, papers and architectures) across six major categories of token reduction , I identified the 10 top methods that deliver real production impact. I evaluated each one against four dimensions: Peak Gain (maximum efficiency under ideal conditions), Quality Impact (how much output quality is affected), Production Readiness (can you deploy today or is it research-only), and Best Use Case (which workloads get the highest ROI). This Review lets you quickly match methods to your constraints-whether you’re optimizing for cost, latency, memory, or all three.

This post breaks down all 10 methods. A quick reference table is available at the bottom.

(Code examples and implementation are comming soon)

The lowest-hanging fruit. If your application makes repeated API calls with shared system prompts, few-shot examples, or static context, you're paying full price for tokens the model has already seen.

Prompt caching stores and reuses frequently accessed context between API calls. Anthropic reports up to 90% cost reduction and 85% latency reduction for long prompts. A January 2026 evaluation found 41–80% cost reduction and 13–31% TTFT improvement across providers for agentic tasks. Cached input tokens are roughly 10x cheaper than regular input tokens for both OpenAI and Anthropic.

Best for: Applications with repeated system prompts, tool definitions, or shared RAG context across requests.

This is the rare "free lunch" in ML optimization. A smaller, faster draft model proposes multiple tokens at once, and the larger target model verifies them in parallel. Because verification is mathematically equivalent to direct generation, output quality is provably identical.

EAGLE-3 reaches 3–6.5x speedup while Medusa achieves 2.2–3.6x. The key advantage: you get pure latency improvement with zero accuracy trade-off. For any latency-sensitive application, this should be your first deployment after caching.

Best for: Real-time applications (chatbots, code completion, interactive agents) where latency directly impacts UX.

Microsoft's LLMLingua uses a small language model to detect and remove unimportant tokens from prompts before they reach your target LLM. The compressed prompts work directly with black-box models like GPT-4 and Claude-no fine-tuning required.

LLMLingua-2 achieves 2x–5x compression while being 3x–6x faster than the original, with end-to-end latency improvement of 1.6x–2.9x. The recommended pattern: apply LLMLingua-2 for dynamic RAG contexts where documents change per request, while using caching for static elements.

Best for: RAG applications with large retrieved contexts, long-document summarization, multi-document QA.

Reasoning models like o1 and DeepSeek-R1 generate verbose chain-of-thought that can consume thousands of output tokens-the most expensive component of API costs. TokenSkip trains models to skip less critical reasoning tokens while maintaining accuracy.

This is critical for the growing class of applications using reasoning models. If your output tokens are your biggest cost driver, CoT compression delivers outsized ROI.

Best for: Applications using reasoning models (o1, DeepSeek-R1) where output tokens dominate costs.

As context windows expand to 128K+ tokens, the KV-cache becomes the dominant memory bottleneck. NVIDIA's RocketKV (ICML 2025) uses a two-stage compression approach achieving up to 400x compression with negligible accuracy loss on long-context tasks.

The Heavy-Hitter Oracle (H2O) retains only 20% of KV cache tokens identified as "heavy hitters" based on attention scores, improving throughput by up to 29x. These methods are most impactful for long-context scenarios where KV cache dominates memory.

Best for: Self-hosted models with long-context workloads (document analysis, code generation, multi-turn conversations).

Standard transformer attention computes relationships between all token pairs-O(n²) complexity that becomes prohibitive at long contexts. MInference identifies dynamic sparse patterns in attention heads, achieving 10x speedup for pre-filling on 1M token contexts while maintaining accuracy within 0.01% of FlashAttention.

FlashAttention from Dao AI Lab provides IO-aware exact attention computation that's 5–9x faster and uses 5–20x less memory than standard attention-and it's now a standard component in most inference stacks.

Best for: Long-context inference (document processing, multi-turn conversations, code repositories).

Token pruning removes uninformative tokens during the forward pass. LazyLLM dynamically selects token subsets during both prefilling and decoding without fine-tuning, achieving 2.34x prefilling speedup on LLaMA 2 7B.

SlimInfer (AAAI 2026) exploits information diffusion across layers to prune less critical tokens, achieving 2.53x time-to-first-token speedup and 1.88x end-to-end latency reduction on LLaMA3.1-8B. These are particularly attractive because they require no model retraining.

Best for: Self-hosted models where you control the inference stack and need latency improvements without retraining.

Rather than discarding tokens, merging combines similar tokens using a lightweight bipartite matching algorithm. Meta AI's ToMe reduces tokens by up to 60% while speeding up generation by 2x and reducing memory by 5.6x-and it's completely training-free.

The key advantage over pruning: merged tokens retain combined semantic content rather than losing information entirely. The speed-up also stacks with efficient implementations like xFormers.

Best for: Vision-language models, diffusion models, and scenarios where information preservation is critical.

While KV-cache eviction removes tokens, quantization reduces the precision of what remains. AQUA-KV optimizes KV-cache quantization using Q/K attention dynamics, enabling extreme compression (2-bit or lower) with negligible accuracy degradation.

The Palu method uses low-rank projection to decompose KV-cache, achieving 91.1% of original accuracy at INT2-level compression. These methods stack with eviction strategies for compound memory savings.

Best for: Memory-constrained deployments, edge inference, maximizing batch size on fixed GPU budget.

At the research frontier, soft-prompt methods compress entire documents into a handful of learned embedding vectors. The 500xCompressor achieves 480x compression while retaining 62.26% of the performance of using the full original context.

Gist tokens take a similar approach, compressing arbitrary prompts into compact learned representations. While not yet production-ready for most use cases, these methods represent the theoretical ceiling of token reduction.

Best for: Research teams exploring extreme efficiency, applications with massive static knowledge bases that can tolerate some accuracy loss.

Use this table to prioritize based on your constraints:

Token efficiency isn't a single optimization-it's a strategy. The teams getting the best results aren't choosing between these methods; they're layering them. Prompt caching for static context, LLMLingua-2 for dynamic inputs, and KV-cache optimization at the serving layer compound savings multiplicatively.

Start with the zero-risk methods this week. You'll likely see 50%+ cost reduction before you even touch your model serving infrastructure. Then layer in preprocessing and inference optimizations as your scale demands.

The research landscape is moving fast-with over 240 papers published in this space, new techniques emerge monthly. But the 10 methods in this playbook represent the proven, practical core that every LLM builder should have in their toolkit.

[1] Awesome Token Compression & Reduction — github.com/ZLKong/awesome-token-compression-reduction

[2] Anthropic Token Saving Updates — anthropic.com/news/token-saving-updates

[3] Speculative Decoding (Wikipedia) — en.wikipedia.org/wiki/Speculative_decoding

[4] LLMLingua: Compressing Prompts for LLMs — github.com/microsoft/LLMLingua

[5] LLMLingua-2: Data Distillation for Prompt Compression (ACL 2024) — arxiv.org/abs/2403.12968

[6] RocketKV: KV-Cache Compression (ICML 2025) — arxiv.org/abs/2502.14051

[7] MInference: Million-Token Inference — github.com/microsoft/MInference

[8] SlimInfer: Token Pruning (AAAI 2026) — arxiv.org/abs/2508.06447

[9] Token Merging (ToMe) — arxiv.org/abs/2303.17604

[10] TokenSkip: Chain-of-Thought Compression — huggingface.co/papers/2502.12067

[11] Compressed CoT Reasoning — arxiv.org/abs/2505.08392

[12] Token Reduction Survey (arXiv 2025) — arxiv.org/abs/2505.18227

[13] LongLLMLingua — microsoft.com/en-us/research/project/llmlingua

[14] KV-Cache Compression Survey — arxiv.org/abs/2508.06297

[15] H2O: Heavy-Hitter Oracle (NeurIPS 2023) — deepai.org/publication/h-2o-heavy-hitter-oracle

[16] KV-Cache Slimming — emergentmind.com/topics/kv-cache-slimming

[17] DynamicViT — github.com/raoyongming/DynamicViT

[18] LazyLLM: Dynamic Token Selection — arxiv.org/abs/2407.14057

[19] Token Merging (OpenReview) — openreview.net/forum?id=JroZRaRw7Eu

[20] K-Token-Merging — github.com/shsjxzh/K-Token-Merging

[21] Token Merging in Transformers — github.com/huggingface/transformers/issues/30292

[22] Prompt Caching Evaluation (2026) — arxiv.org/abs/2601.06007

[23] Prompt Caching Guide — ngrok.com/blog/prompt-caching

[24] TokenSkip Full Paper — arxiv.org/abs/2502.12067

[25] EAGLE-3: Speculative Decoding — arxiv.org/abs/2601.03043

[26] FlashAttention — github.com/Dao-AILab/flash-attention

[27] AQUA-KV: KV-Cache Quantization — arxiv.org/abs/2501.19392

[28] Palu: Low-Rank KV Projection — arxiv.org/abs/2407.21118

[29] 500xCompressor — github.com/ZongqianLi/500xCompressor

[30] Prompt Compression Survey (NAACL 2025) — github.com/ZongqianLi/Prompt-Compression-Survey