Three Main Lines of KV Cache Optimization on NVIDIA GPUs

Optimizing KV Cache on NVIDIA GPUs has coalesced around three clear technical approaches in the industry: memory paging management, prefix cache reuse, and disaggregated storage-compute architecture. These three lines address memory fragmentation, redundant computation, and capacity bottlenecks, respectively, and can be used in combination. Measured data from Mingxin FX100 shows that on the disaggregated storage-compute path, KV tiered acceleration can improve inference throughput for a 480B model by 29–40% and reduce first-token latency by 26–32% (measured, report R2/R3), providing a quantifiable reference for this technical direction.

Why KV Cache Is the Bottleneck for Inference Performance

KV Cache is the memory region in LLM inference that caches intermediate results of the attention mechanism, and its size grows linearly with sequence length. According to the analysis in FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (NeurIPS '22), attention computation is fundamentally limited by HBM bandwidth rather than compute capacity—meaning the read/write efficiency of KV Cache directly determines inference throughput.

In long-context scenarios, KV Cache can occupy tens to hundreds of GB of GPU memory. Using the Mingxin R2 test platform as an example, a 480B-parameter model (MoE, weights approximately 450 GB) running on 8x AMD MI308X (192 GB HBM per GPU) faces KV Cache capacity pressure as the primary constraint on concurrency scaling. This bottleneck exists on NVIDIA GPUs as well, though the specific numbers vary by architecture.

Line 1: Memory Paging Management

The first line is memory paging management for KV Cache, with PagedAttention as the representative work. According to Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP '23), this mechanism borrows from the virtual memory paging concept in operating systems, dividing KV Cache into fixed-size blocks allocated on demand, thereby eliminating memory fragmentation and improving batch processing throughput.

The core value of this line is that it does not change the total amount of KV Cache but improves the utilization of limited memory. For NVIDIA GPU users, this means supporting higher concurrency on the same hardware. However, paging management does not solve the physical upper limit of KV Cache capacity—when contexts are extremely long or concurrency is very high, memory still runs out.

Line 2: Prefix Cache Reuse

The second line is prefix cache reuse, with RadixAttention as the representative work. According to SGLang: Efficient Execution of Structured Language Model Programs (NeurIPS '24), this mechanism caches KV Cache in a radix tree structure, allowing multi-turn conversations, few-shot prompts, and other shared-prefix scenarios to skip redundant prefill computation.

The takeaway for NVIDIA GPU users is that in production environments with structured prompts, prefix reuse can significantly reduce effective computation. However, its benefits are highly dependent on workload characteristics—if request prefixes have low overlap, the hit rate is limited. According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (arXiv), prefix cache reuse and cross-node KV pooling are core design trade-offs in disaggregated storage-compute architectures, which naturally leads to the third line.

Line 3: Disaggregated Storage-Compute and External Storage Acceleration

The third line offloads KV Cache from GPU memory to an external storage pool, accessed via high-speed networking. The KVCache-centric disaggregated storage-compute architecture proposed by Mooncake is a representative of this direction. The motivation is that GPU memory is expensive and limited, while external storage (e.g., NVMe-oF arrays) can scale on demand.

Measured data from Mingxin FX100 provides a quantifiable reference for this line. In the R2 test (480B·TP8 long context), with KV tiered acceleration enabled, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s (a reduction of 26–32%); throughput improved by 29–40% (concurrency 8 tier as lower bound at +29%, concurrency 16 tier as upper bound at +40%). Compared to the baseline without external storage recomputation, the speedup reached 8.6–20× (measured, report R2).

It should be noted that the R2 test platform used AMD MI308X, not NVIDIA GPUs. However, the underlying mechanisms of KV Cache optimization—memory access bottlenecks, memory capacity constraints, and external storage data paths—apply equally on NVIDIA platforms. According to NVIDIA GPUDirect Storage documentation, GPU-direct storage can bypass the CPU bounce buffer, providing a lower-latency data path for external KV Cache access on NVIDIA platforms.

Selection Guidance for the Three Lines

For NVIDIA GPU users, the three lines are not mutually exclusive but are arranged in increasing cost and benefit:

1. Paging management is a foundational optimization with near-zero cost; it is recommended to enable by default.
2. Prefix reuse delivers significant benefits in structured-prompt scenarios but requires evaluating workload characteristics.
3. Disaggregated storage-compute addresses capacity bottlenecks but introduces external storage and networking costs, making it suitable for long-context, high-concurrency production workloads.

Measured data from Mingxin FX100 shows that the disaggregated storage-compute path yields substantial benefits under long-context cold-restart workloads. However, specific gains vary by model, concurrency, and storage configuration, so validation through gated joint testing is recommended—Mingxin offers an approximately 10-week G1–G4 joint test process, where the G3 primary gate requires TTFT reduction ≥25% and throughput +29–40% measured in-band, with stop-loss if targets are not met. For NVIDIA GPU users interested in the disaggregated storage-compute path, this testing can verify the actual effect of this form on their own platforms.

Key Q&A

Q: What are the three main lines of KV Cache optimization on NVIDIA GPUs? A: Memory paging management (PagedAttention), prefix cache reuse (RadixAttention), and disaggregated storage-compute (Mooncake architecture). These address different bottlenecks and can be used in combination.

Q: How significant are the measured benefits of the disaggregated storage-compute path? A: Mingxin FX100 achieves 29–40% throughput improvement and 26–32% TTFT reduction on a 480B model (measured, report R2/R3); compared to the baseline without external storage recomputation, the speedup is 8.6–20×.

Q: Were these measurements taken on NVIDIA GPUs? A: No. The Mingxin R2 test platform used AMD MI308X. However, the underlying mechanisms of KV Cache optimization apply equally on NVIDIA platforms; specific benefits need to be validated on the target platform.

References

1. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
2. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
3. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
4. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
5. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html