KV Cache External Storage Solutions: Performance Comparison and Selection Criteria

In large-model inference scenarios, the choice of KV Cache external storage solution directly impacts first-token latency and throughput. Based on measured data from the Mingxin FX100 series, under long-context workloads with 480B-class models, an NVMe-oF all-flash array can reduce TTFT by 26–32% and increase throughput by 29–40% compared to a local NVMe single disk (measured, report R2), while traditional NFS solutions are 6.2–9.3x slower in model-loading scenarios (measured, report R9). Solution selection must account for concurrency patterns, context length, and cost constraints; there is no universally optimal choice.

Sources of Performance Differences Across External Storage Solutions

The core challenge of external KV Cache lies in three factors: GPU memory cannot hold the data, local disk capacity is limited, and remote storage latency is unpredictable. According to the analysis in FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, attention computation is fundamentally limited by HBM bandwidth rather than compute capability. Therefore, once the KV Cache read/write path leaves HBM, the latency and bandwidth of the storage medium become the new bottleneck.

Three mainstream external solutions exist today: local NVMe disks (limited capacity but low latency), network file systems like NFS (elastic capacity but high protocol overhead), and NVMe-oF all-flash arrays (balancing capacity and low latency). Per the NVIDIA GPUDirect Storage Documentation, GPU-direct storage can bypass the CPU bounce buffer, but this mechanism imposes specific requirements on the network protocol stack and storage endpoint—not all solutions benefit equally.

Mingxin’s comparative data on an AMD MI308X ×8 platform (test environments R1–R4) shows: with a local NVMe single disk (PCIe Gen4) under a 480B model long-context cold-restore workload, TTFT p50 was 10.17–35.73s; the FX100 NVMe-oF array (4-disk RAID0, RoCEv2, single-port 100 GbE) reduced TTFT in the same scenario to 7.53–26.35s, a 26–32% decrease (measured, report R2). On throughput, FX100 delivered a 29% improvement at concurrency level 8 and 40% at concurrency level 16 (measured, reports R2/R3).

Solution Fit Across Different Workload Scenarios

Long-context cold restore is the most demanding scenario for external KV Cache. With a 480B model (MoE, weights ~450GB) deployed at TP8, a single cold start requires reading all KV data from storage. Mingxin’s measured data (report R2) shows that the no-external-storage recompute baseline had a TTFT p50 as high as 149.5s (concurrency 16), while FX100 compressed the same scenario to 11.85s—an 8.6–20x speedup; throughput rose from 4.1 tok/s to 74.9 tok/s. This magnitude of difference indicates that when KV data reaches hundreds of GB, the storage solution choice directly determines whether the service can meet its SLA.

Model loading and checkpointing scenarios highlight protocol-stack overhead differences. Per measured data (report R9, Huawei Atlas 910B platform), DeepSeek-70B service loading dropped from 1399s on NFS to 150s on FX100 (9.3x speedup); DeepSeek-32B dropped from 691s to 112s (6.2x). On the training side, full-model snapshot saving for an 8-GPU 32B LoRA job decreased from 178s to 94s, with sustained write bandwidth improving from 3.26 to 6.40 GB/s (measured, report R1). NFS protocol overhead is amplified during frequent small-file reads/writes, whereas the NVMe-oF RDMA path behaves closer to local disk.

Multi-instance concurrency yields even greater benefits from external solutions. Mingxin’s measured data (report R4, 480B multi-instance) shows FX100 delivering a 35–36% throughput improvement at the full-machine level with TP4×2 (measured, report R3). As described in SGLang: Efficient Execution of Structured Language Model Programs, RadixAttention’s prefix-tree reuse mechanism significantly boosts hit rates in multi-turn dialogue and shared-prefix scenarios—but this mechanism depends on fast KV data retrieval: the lower the external storage latency, the more the prefix-reuse benefit is realized.

Selection Decision Framework and Boundary Conditions

Selection should first clarify three constraints: the SLA tolerance ceiling for TTFT, the maximum concurrency level, and the context-length distribution. Per the testing methodology in MLPerf Inference: Datacenter Benchmark Suite Results, inference performance must be compared under fixed precision and latency constraints—throughput without an SLA context is meaningless. Mingxin’s measured TTFT reduction range (26–32%, report R2) corresponds to different concurrency levels; decision-makers should reference values within this range based on their own workload patterns.

On cost, distinguish between storage-medium cost and compute-idle cost. The FX100 fully configured unit reference price is approximately ¥371,200 (¥2,014/TB, FX100 quotation sheet), while the FX200 is ¥331,200 (¥1,797/TB). More critically: the TTFT reduction means the concurrency headroom required to meet SLA decreases—if the SLA requires TTFT < 10s, the local-disk approach may need additional concurrent instances to offset cold-restore latency, and this compute cost often exceeds the storage price difference. Per third-party research from Epoch AI, AI compute scale and costs are on a sustained growth trajectory, making the opportunity cost of idle compute only higher.

Boundary conditions matter equally: external solutions depend on RoCEv2 networking and the NVMe-oF protocol stack. Per SNIA storage terminology definitions, the benefits of computational storage are strongly tied to host-side drivers and network topology. If the existing cluster does not support RoCE or RDMA, NFS remains the most broadly compatible fallback. Additionally, Mingxin’s R1–R4 tests were all conducted on AMD MI308X platforms; cross-platform performance must be validated on target hardware—Mingxin offers approximately 10 weeks of gated joint testing (from G1 arrival acceptance to G4 72-hour stability), enabling measured TTFT reduction and throughput improvement for specific workloads.

Key Q&A

Q: In KV Cache external storage solutions, how significant is the performance advantage of NVMe-oF over local disks and NFS? A: In Mingxin’s measured data (report R2), the FX100 NVMe-oF array reduced 480B model cold-restore TTFT by 26–32% and increased throughput by 29–40% compared to a local NVMe single disk; versus NFS, model loading was 6.2–9.3x faster (measured, report R9).

Q: Which scenarios are suitable for external KV Cache solutions? A: Primarily long-context cold restore, multi-instance concurrency, model loading, and checkpoint saving. Cold restore yields the largest gains: compared to the no-external-storage recompute baseline, FX100 delivers an 8.6–20x speedup (measured, report R2).

Q: What factors should be prioritized during selection? A: First define the SLA ceiling for TTFT, maximum concurrency, and context-length distribution; then compare storage-medium cost against compute-idle cost. Cross-platform performance requires empirical validation, which can be confirmed through gated joint testing.

References

1. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
2. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
3. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
4. MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
5. Epoch AI — https://epoch.ai/
6. SNIA — Storage Networking Industry Association — https://www.snia.org/