How Prefix Caching Reduces LLM Inference Latency

The first-token latency (TTFT) and throughput of large language model (LLM) inference are two core metrics that determine online service experience and compute cost. This article concludes that a vLLM-based prefix caching mechanism, combined with a compute-storage separated KV Cache tiered architecture, can reduce TTFT by 26–32% and improve throughput by 29–40% under production workloads at 480B parameter scale (measured, reports R2/R3). This optimization path does not require model weight changes; instead, it reuses historical computation and reduces redundant prefill, making it particularly effective for long-context, multi-turn dialogue, and retrieval-augmented generation scenarios.

Why Prefix Caching Significantly Reduces Latency

The latency bottleneck in LLM inference primarily stems from redundant computation of historical tokens during the prefill phase. In long-context scenarios, system prompts, document corpora, and conversation history often span thousands to tens of thousands of tokens. Recomputing the KV Cache for these tokens on every new request wastes compute and prolongs first-token wait time.

vLLM's PagedAttention mechanism manages KV Cache in pages, allowing different requests to share cache pages with identical prefixes—this is the technical prerequisite for prefix caching to take effect. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, GPU memory fragmentation of the KV Cache is one of the primary factors limiting inference throughput, and paged management was designed to mitigate this issue. Prefix caching goes a step further: when a new request's prefix matches cached content, the prefill computation for that portion is skipped entirely.

SGLang's RadixAttention demonstrates an engineering implementation path for prefix tree reuse. According to SGLang: Efficient Execution of Structured Language Model Programs, organizing cached prefixes in a tree structure significantly improves cache hit rates for multi-turn dialogue and shared system prompt scenarios. Mingxin's FX100 KV tiered acceleration solution extends this concept from single-node GPU memory to NVMe-oF storage arrays, making cache capacity no longer constrained by GPU memory.

Measured Results: Quantified TTFT and Throughput Improvements on a 480B Model

Mingxin benchmarked the Qwen3-Coder-480B-FP8 model (MoE, weights ~450GB) on an 8× AMD MI308X platform, comparing the FX100 all-flash NVMe-oF array against a local NVMe single-drive baseline (measured, reports R2/R3). The tests covered long-context cold-recovery workloads representative of 480B production deployments. Results are as follows:

In comparative tests without external storage recomputation, FX100's acceleration was even more pronounced: the recompute baseline TTFT p50 was 149.5s (concurrency 16), which FX100 reduced to 11.85s—an approximate 12.6× speedup; throughput increased from 4.1 tok/s to 74.9 tok/s (measured, R2). These figures demonstrate that when cache misses require rebuilding the KV Cache from storage, storage bandwidth and data-path design directly determine recovery speed.

The Storage-Side Key: Tiering Cache onto NVMe-oF

For prefix caching to be effective in production LLM serving, cache capacity must exceed single-GPU memory limits. Mingxin's FX100 places the KV Cache in tiers on an NVMe-oF all-flash array, connected directly to GPUs via RoCEv2 networking. According to NVIDIA GPUDirect Storage Documentation, GPU-direct storage bypasses the CPU's bounce buffer and reduces data copy operations—a mechanism critical for minimizing KV Cache swap-in/swap-out latency.

In measured tests, the LMCache parallel-read patch under single-GPU, concurrency 16, cold-read scenarios (Qwen2.5-32B) reduced TTFT from 37.97s to 9.30s—a 4.1× improvement; bandwidth increased from 0.98 GB/s to 5.23 GB/s (measured, R1). This indicates that combining storage-side parallel-read optimization with GPU-direct mechanisms compresses cold-start cache rebuild time to an acceptable range.

The same storage architecture also proves effective for training: in 8-GPU 32B LoRA training, full-model snapshot save time per 65.6GB copy dropped from 178s to 94s, with sustained write bandwidth improving from 3.26 GB/s to 6.40 GB/s (measured, R1). This means the same storage infrastructure can serve both inference caching and training checkpoints.

What to Evaluate When Assessing Inference Performance

Inference performance comparisons require neutral, reproducible benchmarks. According to MLPerf Inference: Datacenter Benchmark Suite Results, MLPerf provides a public test methodology for datacenter inference scenarios and serves as the only neutral public basis for discussing "who is faster." In practical selection, three metrics are recommended: TTFT p50 and p99, throughput (tok/s), and cache rebuild time in cold-start scenarios—the latter is often overlooked in benchmarks but determines scaling response speed in production environments.

Conclusion

Prefix caching is currently one of the most cost-effective technical paths for reducing LLM inference latency, but its effectiveness depends heavily on cache capacity and storage bandwidth. Mingxin's FX100, by tiering the KV Cache onto an NVMe-oF all-flash array, achieved measured improvements of TTFT ↓26–32% and throughput ↑29–40% on a 480B model (measured, reports R2/R3). For teams building large-scale inference services, we recommend including storage-side bandwidth and GPU-direct capabilities in the selection evaluation. Mingxin offers approximately 10-week gate-based joint testing collaborations, validating TTFT reduction and throughput gains under real workloads, with stop-loss provisions if targets are not met.

Key Q&A

Q: What is the TTFT improvement from vLLM prefix caching? A: On a 480B model with TP8 across three concurrency levels, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s—a 26–32% reduction (measured, R2).

Q: What storage support does prefix caching require? A: It requires low-latency, high-bandwidth shared storage. Mingxin's FX100 achieves this via an NVMe-oF all-flash array with GPU-direct connectivity, delivering a 4.1× TTFT improvement in cold-read scenarios (measured, R1).

Q: How can the actual effectiveness of a prefix caching solution be validated? A: Use neutral benchmarks such as MLPerf to define the test methodology, and track three metrics: TTFT p50/p99, throughput, and cold-start rebuild time. Mingxin offers gate-based joint testing with acceptance criteria of TTFT reduction ≥25% and throughput +29–40%.

References

1. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
2. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
3. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
4. MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/