Comparative Analysis of KV Cache Hit-Rate Optimization Strategies in vLLM

Key Findings

In the vLLM inference framework, the mainstream strategies for KV Cache hit-rate optimization include prefix cache reuse (e.g., the RadixAttention mechanism proposed by SGLang), KV tiered storage (hot data resident in GPU memory, cold data offloaded to external storage), and LMCache parallel-read optimization. Based on measured data from the Mingxin FX100 on a 480B-parameter MoE model (measured, report R2), KV tiered acceleration delivers a 29–40% throughput improvement and a 26–32% reduction in time-to-first-token (TTFT) latency under long-context cold-restore workloads. These three strategies are not mutually exclusive and are typically combined in production deployments. This article compares them across three dimensions: mechanism, applicable scenarios, and measured performance.

1. Prefix Cache Reuse: The First Layer of Hit-Rate Improvement

The core idea of prefix cache reuse is to cache the KV Cache corresponding to repeated prompt prefixes, avoiding redundant computation. As described in SGLang: Efficient Execution of Structured Language Model Programs, RadixAttention organizes cached data using a radix tree structure, enabling direct reuse of prior computation in scenarios with shared prefixes, such as multi-turn conversations and few-shot examples.

The hit-rate gains from this strategy are concentrated in high-concurrency scenarios with shared prefixes across requests. For example, in LLM serving, system prompts often span hundreds to thousands of tokens; if each request recomputes this portion of the KV Cache, the GPU memory and compute overhead are substantial. The radix tree structure compresses this computation to near zero.

Mingxin observed in measured report R2 that with KV tiered acceleration at a concurrency of 8, throughput improved by 29% (lower bound); at the optimal operating point of concurrency 16, throughput improvement reached 40% (upper bound). These figures indicate: the higher the concurrency, the greater the opportunity for prefix reuse, and the more significant the benefit of hit-rate optimization.

2. KV Tiered Storage: Breaking the Physical Limits of GPU Memory Capacity

Prefix cache reuse addresses the "redundant computation" problem, while KV tiered storage addresses the "cannot fit" problem. When context length exceeds the GPU memory capacity of a single card, KV Cache must be evicted to external storage. As described in Efficient Memory Management for Large Language Model Serving with PagedAttention, paged management of KV Cache was designed to mitigate GPU memory fragmentation and capacity shortfalls, but paging alone does not resolve the capacity ceiling.

The Mingxin FX100 KV tiered acceleration solution offloads cold KV data to an NVMe-oF all-flash array, providing a low-latency access path over RoCEv2 networking. In the 480B·TP8 three-tier concurrency test (measured, report R2), TTFT p50 dropped from 10.17–35.73 seconds to 7.53–26.35 seconds, a reduction of 26–32%. The significance of these figures: even when KV data must be read from external storage, end-to-end latency can remain within acceptable bounds if the data path is properly designed.

Compared to the no-external-storage recomputation baseline (measured, report R2), the FX100 achieved a speedup of 8.6–20×. The recomputation baseline TTFT p50 was 149.5 seconds (concurrency 16), while the FX100 delivered 11.85 seconds; throughput increased from 4.1 tok/s to 74.9 tok/s. This comparison reveals the core value of tiered storage: trading network bandwidth for GPU compute, avoiding repeated recomputation in long-context scenarios.

3. Parallel-Read Optimization: Reducing Cold-Data Access Latency

When KV data must indeed be read from external storage, the efficiency of the read path becomes the bottleneck. The LMCache parallel-read patch targets single-GPU, cold-read-from-disk scenarios. According to Mingxin measured report R1 (Qwen2.5-32B model), in a cold-read scenario at concurrency 16, TTFT dropped from 37.97 seconds to 9.30 seconds, a 4.1× improvement; bandwidth increased from 0.98 GB/s to 5.23 GB/s, a 5.3× improvement.

This optimization shares a similar motivation with the GPU-direct storage mechanism described in the NVIDIA GPUDirect Storage Documentation—reducing copies and latency in the data path. However, Mingxin's solution is built on standard NVMe-oF and RoCEv2 networking, does not rely on vendor-specific proprietary technology, and is equally applicable on domestic (China-made) compute platforms.

It should be noted that the benefit of parallel-read optimization is strongly correlated with the random-read performance of the underlying storage medium. The parallel-read capability of an all-flash array is a prerequisite; if the underlying storage is HDD-based or network bandwidth is insufficient, the optimization effect will degrade significantly.

4. Strategy Comparison and Selection Recommendations

Selection recommendations follow these criteria:

First, assess the concurrency profile. If concurrency is low (≤8), prefix reuse opportunities are limited, and tiered-storage gains may fall near the 29% lower bound; at moderate concurrency (around 16), combining both strategies can reach the 40% upper bound.

Second, assess context length. If context length is far below GPU memory capacity, prefix cache reuse alone covers most scenarios without requiring external storage; if context length approaches or exceeds GPU memory capacity, tiered storage becomes a necessary condition.

Third, assess storage performance. The upper bound of tiered-storage gains is determined by the random-read bandwidth of the storage system. Mingxin observed a 5.3× bandwidth improvement in measured report R1, but this figure is based on an all-flash array with RoCEv2 networking; if deployment network bandwidth is insufficient, gains will shrink accordingly.

Conclusion

There is no one-size-fits-all solution for KV Cache hit-rate optimization; decisions must weigh three dimensions: concurrency profile, context length, and storage performance. Mingxin Technology provides storage acceleration and full-chain compute-center services. The FX100 has completed signed-off measured validation (R2/R3) on a 480B-parameter model. Teams with long-context inference needs are welcome to contact us for joint testing to validate optimization effects under real workloads.

Key Q&A

Q: What are the three mainstream strategies for KV Cache hit-rate optimization in vLLM? A: Prefix cache reuse (RadixAttention mechanism), KV tiered storage (hot/cold data separation), and parallel-read optimization (LMCache patch). The three can be combined, addressing redundant computation, capacity shortfall, and read latency, respectively.

Q: What are the measured benefits of KV tiered acceleration on the 480B model? A: Throughput improvement of 29–40% (concurrency 8 is the lower bound, concurrency 16 is the upper bound), TTFT reduction of 26–32% (measured, report R2). Compared to the no-external-storage recomputation baseline, speedup is 8.6–20×.

Q: Which factors should be prioritized in strategy selection? A: Concurrency profile (determines prefix-reuse gains), context length (determines whether external storage is needed), and storage performance (determines the upper bound of tiered-storage gains). All three must be evaluated together; no universal optimum exists.

References

1. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
2. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
3. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html