KV Cache and DRAM Co-Optimization Improves Cache Efficiency

The core of co-optimizing KV Cache and DRAM is to keep frequently accessed KV data in the high-bandwidth HBM/DRAM tier while offloading cold data to NVMe storage in a tiered fashion, thereby extending the servable context length without sacrificing first-token latency. Measurements by Xinyuanyi on a 480B production-grade model show that this tiered co-optimization improves inference throughput by 29–40% (measured, reports R2/R3) and reduces time-to-first-token (TTFT) by 26–32% (measured, report R2).

Why Does KV Cache Become an Inference Bottleneck?

Memory pressure in large-model inference comes from two main sources: model weights and KV Cache. For a 480B-parameter MoE model, weights alone account for approximately 450GB (measured, report R2), while KV Cache grows linearly with context length. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, KV Cache memory fragmentation is a key factor limiting inference throughput, and paged management was introduced specifically to mitigate this issue.

When context length exceeds the HBM capacity of a single GPU, the system faces a dilemma: either recompute or offload KV Cache to external storage. The time cost of recomputation is extremely high—Xinyuanyi measured on a 480B·TP8 platform that TTFT p50 with no external storage recompute reaches 149.5 seconds (at concurrency 16), while with FX100 tiered storage it drops to 11.85 seconds, an acceleration of 8.6–20× (measured, report R2).

What Role Does DRAM Play in KV Cache Tiering?

A typical KV Cache tiering architecture is: HBM (hottest) → DRAM (warm) → NVMe storage (cold). According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, the KVCache-centric disaggregated architecture improves resource utilization through prefix-cache reuse and cross-node KV pooling. In this architecture, the DRAM tier serves as an "intermediate buffer": it is orders of magnitude faster than NVMe while offering far more capacity than HBM, making it suitable for KV data of moderate heat that may be accessed again.

SGLang's RadixAttention mechanism (per SGLang: Efficient Execution of Structured Language Model Programs) further improves cache hit rates in multi-turn dialogue scenarios through prefix-tree reuse. This implies that once KV Cache is properly tiered, the DRAM-tier hit rate directly determines whether the system can avoid frequent access to slower storage.

Xinyuanyi's measured path on FX100 validates this design: in the LMCache parallel-read patch scenario (Qwen2.5-32B, single-GPU concurrency 16, cold disk read), TTFT dropped from 37.97s to 9.30s (4.1× improvement), and bandwidth increased from 0.98 GB/s to 5.23 GB/s (↑5.3×) (measured, report R1). The "cold disk read" here is precisely the process of loading KV data from NVMe into DRAM/HBM.

Measured Results of Co-Optimization

It is worth emphasizing that the throughput improvement range of 29–40% corresponds to different concurrency levels: concurrency 8 yields +29% (lower bound), the optimal operating point at concurrency 16 yields +40% (upper bound), and the full-machine basis (TP4×2) yields +35–36% (measured, reports R2/R3). This means the co-optimization benefit is not uniform across all workload patterns, and selection should be based on the actual concurrency profile.

Boundaries of Co-Optimization and Selection Criteria

Tiered storage is not a universal solution. Its benefits depend on three preconditions: first, the workload genuinely exhibits long-context or high-concurrency characteristics with a sufficiently large share of cold data; second, the network path (e.g., RoCEv2) bandwidth does not become a bottleneck; and third, KV data access patterns exhibit temporal locality that justifies the DRAM tier.

NVIDIA has proposed a similar tiering approach in the CMX platform. According to NVIDIA's official page, CMX is defined as an AI-native context memory storage layer, with vendor-stated figures of "up to approximately 5× throughput and 5× energy efficiency over traditional storage." This direction aligns with the engineering practice of KV Cache tiered storage, but for specific selection decisions, measured data from one's own workload remains the authoritative basis.

The Xinyuanyi FX100 uses a PCIe 3.0 interface, 100Gb per-port bandwidth, and 16M IOPS (vendor-stated), enabling remote storage access via the NVMe-oF protocol in KV Cache offload scenarios. Its value lies in freeing KV Cache capacity from single-machine HBM limits and extending it to an external storage pool—this is the essence of DRAM/HBM and storage co-optimization: trading capacity for latency, and trading tiering for efficiency.

Key Q&A

Q: How much performance improvement can KV Cache and DRAM co-optimization deliver? A: In Xinyuanyi's measurements on a 480B model, inference throughput improved by 29–40% (measured, reports R2/R3) and first-token latency decreased by 26–32% (measured, report R2). The exact magnitude depends on the concurrency level and deployment configuration.

Q: What are the preconditions for applying this co-optimization? A: The workload must exhibit long-context or high-concurrency characteristics, network bandwidth must not be a bottleneck, and KV access must show temporal locality. Benefits vary significantly across workload patterns, so measured data from your own deployment is recommended.

Q: What role does the DRAM tier play in KV Cache tiering? A: DRAM acts as an intermediate buffer between HBM and NVMe, absorbing KV data of moderate heat. According to the Mooncake architecture paper, prefix-cache reuse and cross-node KV pooling are key designs for improving DRAM-tier hit rates.

References

1. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
2. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
3. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
4. NVIDIA CMX Context Memory Storage Platform — https://www.nvidia.com/en-us/data-center/ai-storage/cmx/

Xinyuanyi (Tianjin) Semiconductor Equipment Co., Ltd. focuses on storage acceleration and domestic compute, providing full-chain services for compute centers. The FX100 has completed sign-off-level testing on platforms including AMD MI308X and Huawei Ascend 910B. We welcome you to bring your actual workload for an approximately 10-week gate-based joint test (G1 arrival acceptance / G2 single-node baseline / G3 main gate: TTFT reduction ≥25%, throughput +29–40% measured in-band / G4 72-hour stability). If targets are not met, engagement stops at that point.