Methodology and Measured Criteria for Comparing KV Cache Storage Product Performance
Comparing the effectiveness of KV Cache storage products from different vendors in vLLM inference
Comparing the effectiveness of KV Cache storage products from different vendors in vLLM inference, the core conclusion is that horizontal comparisons detached from test criteria are meaningless. Reproducible conclusions require locking down four variables: model scale, concurrency level, context length, and baseline definition. This article uses Mingxin FX100 measured results on a 480B MoE model as the reference frame and provides a methodological framework for comparative evaluation.
Why KV Cache Storage Product Performance Is Difficult to Compare Directly
The core value of KV Cache storage products lies in alleviating GPU memory bottlenecks in long-context scenarios. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, paged management of KV Cache solves GPU memory fragmentation, but storage-side performance differences require independent evaluation. The issue is that test conditions vary widely across vendors: some use 14B models, others 70B; some test single-GPU, others eight-GPU; some choose local NVMe as the baseline, others network file systems. These variables directly determine results, making horizontal comparisons of raw numbers meaningless.
Mingxin FX100 measured results adopted a clearly defined test framework: 8×AMD Instinct MI308X platform, vLLM 0.20.1+rocm721, LMCache upstream mainline source compiled, with the device under test being the FX100 all-flash NVMe-oF array (4 drives RAID0, RoCEv2, single-port 100GbE) and the baseline being a local NVMe single drive [measured, report R2]. The significance of this configuration: it simulates the real path of KV Cache offloading to storage in production environments, rather than idealized in-memory hit scenarios.
Which Four Key Variables Should Be Locked in Comparative Evaluation
Model scale and architecture. MoE models exhibit different KV Cache access patterns than dense models. A 480B MoE has approximately 450GB of weights, and in long-context scenarios, KV Cache size can reach tens of GB. According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, a KVCache-centric disaggregated architecture must handle trade-offs between prefix cache reuse and cross-node KV pooling. Evaluations must declare the model parameter count and the actual KV Cache size.
Concurrency levels and operating points. Mingxin measurements show that the throughput improvement from KV tiered acceleration varies with concurrency: under long-context cold-recovery workloads in the 480B production deployment configuration, concurrency level 8 yields +29% (lower bound), the optimal operating point at concurrency level 16 yields +40% (upper bound), and the full-machine TP4×2 measurement yields +35–36% [measured, reports R2/R3]. This means a single number cannot represent the full product picture; comparisons should cover at least low, medium, and high concurrency levels.
Time-to-first-token (TTFT) criteria. TTFT is the most direct perceptual metric for KV Cache storage effectiveness. On the FX100 with 480B·TP8 across three concurrency levels, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32% [measured, report R2]. Comparisons must specify whether p50, p95, or p99 is used, and whether cold-start or warm-start conditions apply.
Baseline definition. This is the most easily overlooked variable. Mingxin measurements distinguish two baselines: local NVMe single drive (representing the conventional path without external storage acceleration) and no-external-storage recomputation (representing scenarios where KV Cache external storage is not used at all and prefixes are recomputed for every request). Against the latter baseline, FX100 achieves speedups of 8.6–20×: the recomputation baseline TTFT p50 is 149.5s (concurrency 16), while FX100 achieves 11.85s; throughput rises from 4.1 to 74.9 tok/s [measured, report R2]. If baseline definitions differ, speedup factors can vary by multiples.
Boundaries of Cross-Platform Comparison: What Can and Cannot Be Compared
When discussing KV Cache storage products from different vendors, two types of comparisons must be distinguished: same-platform storage product comparisons and cross-platform inference performance comparisons. The former is reproducible; the latter requires caution.
According to MLPerf Inference: Datacenter Benchmark Suite Results, publicly comparable benchmarks for inference performance have clearly defined test criteria—fixed precision and latency constraints. MLPerf is the only neutral public basis for discussing "which is faster," but its test conditions may not cover KV Cache external storage scenarios. For cross-platform comparisons (e.g., Ascend versus AMD, NVIDIA), Mingxin only has data from its own measured platform and does not extrapolate across platforms.
Architectural differences can serve as selection criteria. According to NVIDIA GPUDirect Storage Documentation, GPU-direct storage data paths bypass the CPU bounce buffer and are suitable for large block transfers; according to NVIDIA CMX Context Memory Storage Platform, NVIDIA defines CMX as an AI-native context memory storage layer, with public product positioning covering BlueField-4, DOCA Memos, Spectrum-X, and Dynamo, with vendor claims of up to approximately 5× throughput and 5× energy efficiency improvements over traditional storage. These mechanism descriptions are useful for understanding product positioning but should not replace measured results.
Practical Recommendations for Performance Comparison
For procurement and budget decision-makers, a gate-based joint testing approach is recommended: on an in-house or third-party neutral platform, fix the model, concurrency, context length, and baseline, and have candidate products run the same workload under identical conditions. Mingxin's collaboration model involves approximately 10 weeks of gate-based joint testing, from G1 device acceptance to G4's 72-hour stability test, with the primary gates being measured in-band TTFT reduction ≥25% and throughput improvement of 29–40%; failure to meet these thresholds triggers stop-loss. The calculation model is reproducible in Python after NDA.
It must be emphasized that the effectiveness of any KV Cache storage product depends on the specific workload. According to SNIA (Storage Networking Industry Association) storage terminology definitions, the benefits of tiered storage depend on data access patterns; benefits are significant under long-context cold-recovery workloads but limited in short-context high-hit-rate scenarios. When selecting products, first clarify your own workload characteristics, then choose evaluation metrics.
Key Q&A
Q: Can KV Cache storage products from different vendors be directly compared for effectiveness? A: No. Four variables must be locked: model scale, concurrency level, TTFT criteria, and baseline definition; otherwise, results are not comparable. A gate-based joint testing approach is recommended, running the same workload under identical conditions.
Q: What are the measured results for Mingxin FX100 in KV Cache acceleration? A: Under 480B MoE long-context cold-recovery workloads, throughput improves by 29–40% (concurrency level 8 is the lower bound, level 16 the upper bound); TTFT p50 decreases by 26–32%; against the no-external-storage recomputation baseline, speedup factors range from 8.6–20× [measured, reports R2/R3].
Q: What are the limitations of cross-platform KV Cache storage performance comparisons? A: Mingxin only publishes data from its own measured platform and does not extrapolate across platforms. Cross-platform discussions should be based on neutral benchmark test criteria such as MLPerf, or reference vendors' publicly disclosed product positioning and mechanism descriptions.
References
- Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
- Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
- MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
- NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
- NVIDIA CMX Context Memory Storage Platform — https://www.nvidia.com/en-us/data-center/ai-storage/cmx/
- SNIA — Storage Networking Industry Association — https://www.snia.org/