NVMe-oF and KV Cache Compatibility: Measured Results and Solutions

Key Findings First

The core compatibility issue between NVMe-oF (NVMe over Fabrics) and KV Cache lies in whether the latency and bandwidth of remote storage access can meet the real-time requirements of KV Cache reads/writes in large-model inference. Measured results on the AMD MI308X platform with the Mingxin FX100 show that using NVMe-oF to carry tiered KV Cache storage, combined with cache reuse mechanisms, improves inference throughput by 29–40% and reduces time-to-first-token (TTFT) by 26–32% [measured, reports R2/R3]. This solution has been validated at 480B parameter scale with TP8 deployment, providing a reproducible engineering path for KV Cache implementation under disaggregated storage-compute architectures.

Compatibility Challenges Between KV Cache and NVMe-oF

In large-model inference, KV Cache capacity requirements grow linearly with context length. For a 480B-parameter MoE model, KV Cache can reach hundreds of GB in long-context scenarios, far exceeding single-GPU HBM capacity [measured, report R2]. Offloading KV Cache to remote storage requires solving two core problems:

First, protocol semantic matching. NVMe-oF implements remote direct memory access over RDMA protocols, with block-level storage semantics as its design goal [RFC 5040]. KV Cache access, however, is highly random—different requests have different prefixes, causing cache hit locations to be scattered. The RadixAttention mechanism proposed by SGLang reuses shared prefixes via a prefix tree, while the Mooncake architecture emphasizes a KVCache-centric disaggregated storage-compute design [SGLang arXiv:2312.07104] [Mooncake arXiv:2407.00079]. These software-level optimizations require the underlying storage protocol to provide low-latency random read capability.

Second, data path efficiency. In traditional storage paths, data must be copied through CPU memory, creating a bounce buffer bottleneck. NVIDIA GPUDirect Storage bypasses the CPU to establish a direct data path between GPU and storage devices, but this mechanism primarily targets locally attached GPU storage [NVIDIA GDS Documentation]. In NVMe-oF scenarios, data traverses the network, resulting in a longer path and more difficult latency control.

Measured data from the Mingxin FX100 shows that under 480B·TP8 with three concurrency levels, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s [measured, report R2]. This result demonstrates that NVMe-oF, when properly configured, can meet the latency budget for KV Cache access.

Tiered Acceleration Solution and Measured Data

The Mingxin FX100 employs a tiered KV Cache storage approach: hot data resides in GPU memory, warm data is stored on local NVMe, and cold data is offloaded to an NVMe-oF array. The test platform consisted of 8×AMD MI308X (192GB HBM per GPU), vLLM 0.20.1+rocm721, and LMCache built from upstream main branch source [measured, reports R1–R4].

The table above shows that throughput improvement varies with concurrency: concurrency 8 represents the lower bound at +29%, concurrency 16 reaches the upper bound at +40%, and the TP4×2 full-node figure is +35–36% [measured, reports R2/R3]. Compared to the no-external-storage recompute baseline (TTFT p50 of 149.5s), the FX100 reduces TTFT to 11.85s and increases throughput from 4.1 to 74.9 tok/s [measured, report R2].

In the LMCache parallel-read patch scenario, single-GPU·concurrency-16·cold-disk-read (Qwen2.5-32B) TTFT dropped from 37.97s to 9.30s, with bandwidth improving from 0.98 to 5.23 GB/s [measured, report R1]. This indicates that NVMe-oF compatibility issues can be effectively mitigated through software patches and protocol optimization.

Key Configurations for Engineering Deployment

A compatible NVMe-oF and KV Cache solution requires attention to the following configuration dimensions:

Network and protocol selection. Testing used RoCEv2 with 100GbE per port. The RDMA protocol provides reliable transport and low-latency guarantees [RFC 5040]. For multi-node deployments, the cross-node KV pooling design in the Mooncake architecture can further reduce cache miss rates [Mooncake arXiv:2407.00079].

Cache policy and storage tiering. PagedAttention's paging management divides KV Cache into fixed-size blocks, reducing GPU memory fragmentation [PagedAttention arXiv:2309.06180]. Combined with LMCache's parallel-read patch, cold data read efficiency can be significantly improved.

File system and block layer tuning. The Linux kernel block layer I/O path and NVMe driver behavior directly impact performance [Linux Kernel Documentation]. Testing used a 4-disk RAID0 configuration (14TB, XFS), leveraging striping to improve parallel read bandwidth [measured, reports R1–R4].

For domestic substitution scenarios, Huawei's CANN heterogeneous computing architecture provides a software stack corresponding to the CUDA ecosystem [CANN-Huawei Ascend Community]. Measured results from Mingxin on the Huawei Atlas 910B platform show model inference loading acceleration of 6.2–9.3× (DeepSeek-32B: 691s→112s; DeepSeek-70B: 1399s→150s) [measured, report R9]. This provides a reference path for KV Cache offloading in domestic IT environments.

Conclusion

The compatibility issue between NVMe-oF and KV Cache is not insurmountable. Through the combination of tiered storage architecture, RDMA protocol optimization, and cache reuse mechanisms, the Mingxin FX100 achieves 29–40% inference throughput improvement and 26–32% TTFT reduction on a 480B model [measured, reports R2/R3]. This solution has been validated on both AMD MI308X and Huawei 910B platforms and is engineering-reproducible. For teams planning to deploy long-context inference services, we recommend validating performance under specific workloads in a joint test environment.

Mingxin offers an approximately 10-week gated joint-test collaboration model (G1 arrival acceptance / G2 single-node baseline / G3 main gate: TTFT reduction ≥25%, throughput +29–40% measured in-band / G4 72-hour stability), with stop-loss if targets are not met. The measurement model is Python-reproducible after NDA. Teams with requirements are welcome to contact us for joint testing and validation.

Key Q&A

Q: What exactly does the NVMe-oF and KV Cache compatibility issue refer to? A: The core issue is whether remote storage access latency and bandwidth can meet the real-time requirements of KV Cache reads/writes. Mingxin's measurements show that through tiered storage and protocol optimization, inference throughput can be improved by 29–40% and TTFT reduced by 26–32% [measured, reports R2/R3].

Q: On which platforms has this solution been validated? A: Both the AMD MI308X platform (8 GPUs, 480B model) and the Huawei Atlas 910B platform (DeepSeek-70B loading acceleration of 9.3×) have measured data [measured, reports R2/R9]. Testing used RoCEv2 networking and NVMe-oF arrays.

Q: What key configurations are required to deploy this solution? A: We recommend using RDMA protocols (RoCEv2), paged management of KV Cache, parallel-read patches, and tuning of the block layer and file system. Specific configurations should be validated per workload during joint testing.

References

1. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
2. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
3. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
4. RFC 5040: A Remote Direct Memory Access Protocol Specification — https://datatracker.ietf.org/doc/html/rfc5040
5. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
6. The Linux Kernel documentation — https://www.kernel.org/doc/html/latest/
7. CANN-Huawei Ascend Heterogeneous Computing Architecture-Ascend Community — https://www.hiascend.com/software/cann