Externalizing KV Cache: How a Storage Layer Can Equivalently Replace 128GB-Class HBM Residency

The HBM residency challenge of KV Cache has escalated from an "engineering optimization" to an "architectural choice." The core conclusion of this article: by implementing tiered management and asynchronous refill of KV Cache through an external storage layer, it is possible to equivalently replace the portion of 128GB-class HBM used for cache residency while preserving inference quality—measured data from Mingxin FX100 under a 480B model long-context workload shows this approach delivers in-band gains of +29–40% in inference throughput and ↓26–32% in TTFT【measured, report R2/R3】. The essence of this path is transforming the problem of "what HBM cannot hold" into "whether storage is fast enough."

Why Does KV Cache Become an HBM Bottleneck?

The HBM pressure in large-model inference does not originate from the weights themselves. Taking a 480B-parameter MoE model as an example, its weights total approximately 450GB and can be distributed across multiple GPUs via tensor parallelism; what truly scales linearly with concurrency and context length is the per-request, independently generated KV Cache. According to the analysis in Efficient Memory Management for Large Language Model Serving with PagedAttention, HBM fragmentation and management overhead of KV Cache are among the core factors constraining serving throughput, and its motivation and quantitative analysis provide the theoretical starting point for paged management. The paper also notes that the throughput gains achieved by vLLM through PagedAttention are bounded by total physical HBM capacity—when cache demand exceeds HBM capacity, trade-offs are unavoidable regardless of paging efficiency.

The conventional solution is "recomputation": discarding part of the KV Cache and recomputing it when needed. In Mingxin's measured R2 results, the no-external-storage recomputation baseline showed a TTFT p50 of 149.5s (at concurrency level 16), compared to FX100's 11.85s—a difference of an order of magnitude【measured, report R2】. The cost of recomputation is multiplied compute idle time and latency degradation, which is nearly unacceptable in long-context, high-concurrency scenarios.

How Does an External Storage Layer "Equivalently Replace" HBM Residency?

The replacement logic does not move the entire KV Cache out of HBM; instead, it tiers the data: hot data remains resident in HBM, while warm/cold data is placed on an external storage layer and refilled on demand via a high-speed NVMe-oF array when GPUs require it. This approach shares its origins with the KVCache-centric disaggregated architecture described in Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving—that paper explores design trade-offs in prefix cache reuse and cross-node KV pooling, with the core insight that KV Cache access patterns (sequential writes, random reads, high prefix reuse) align closely with the capability profile of storage systems.

Mingxin FX100 measured data validates this alignment. Under a 480B·TP8 long-context workload, TTFT p50 across three concurrency levels dropped from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32%【measured, report R2】. More critically, the throughput metrics: concurrency level 8 improved by 29% (lower bound), the optimal operating point at concurrency 16 improved by 40% (upper bound), and the full-machine TP4×2 measurement was +35–36%【measured, reports R2/R3】. These figures imply that, under identical SLA constraints, the concurrency headroom required by the system can be reduced—this is the quantitative meaning of "equivalently replacing HBM."

Measured Data Comparison: External Storage vs. Local NVMe Baseline

For readers to verify item by item, the table below summarizes key comparative data from Mingxin's measured R2 results:

The test platform comprised 8× AMD Instinct MI308X (192GB HBM per GPU), vLLM 0.20.1+rocm721, with the model Qwen3-Coder-480B-FP8 (MoE, weights approximately 450GB). FX100 was configured as a 4-drive RAID0 (14TB, XFS) with single-port RoCEv2 at 100GbE【measured, report R2】.

What Are the Prerequisites and Boundaries of This Approach?

External storage replacing HBM residency is not unconditionally valid; its effectiveness depends on three prerequisites:

First, the storage system must be sufficiently fast. Refill latency directly determines TTFT. Mingxin FX100, in a single-GPU, concurrency-16, cold-read scenario (Qwen2.5-32B) with the LMCache parallel-read patch, reduced TTFT from 37.97s to 9.30s and increased bandwidth from 0.98 to 5.23 GB/s (↑5.3×)【measured, report R1】. If storage bandwidth is insufficient, refill becomes the new bottleneck.

Second, the access pattern must match. The higher the prefix cache reuse rate, the greater the benefit of externalization. The RadixAttention mechanism proposed in SGLang: Efficient Execution of Structured Language Model Programs significantly improves cache hit rates in multi-turn dialogue and shared-prefix scenarios through prefix-tree reuse—this is precisely the workload shape that benefits most from an external storage layer.

Third, the software stack requires deep adaptation. The LMCache parallel-read patch used in Mingxin's measurements is not out-of-the-box; it is a custom optimization tailored to storage characteristics【measured, report R1】. According to NVIDIA's official positioning of the CMX Context Memory Storage Platform, which defines CMX as an AI-native context storage layer and cites vendor figures of "up to approximately 5× throughput / 5× energy efficiency over traditional storage," this indirectly confirms the necessity of hardware-software co-design in this domain.

Conclusion

Externalizing KV Cache is an engineering path validated by measured results: it does not attempt to make HBM "larger" but rather makes storage "fast enough" to absorb the cache overflow. Mingxin has accumulated reproducible test methodologies and measured data in this direction (R1–R9 report series) and offers an approximately 10-week gated joint-testing mechanism (from G1 arrival acceptance to G4 72-hour stability), enabling compute centers with specific needs to validate benefit boundaries on their own workloads. For discussions on adaptation solutions for specific models or concurrency patterns, please contact us for joint testing.

Key Q&A

Q: Can an external storage layer truly replace 128GB-class HBM residency? A: It can replace the portion used for KV Cache caching, not all of HBM. Mingxin's measured R2 results show that under a 480B model long-context workload, the FX100 external storage solution delivers in-band improvements of +29–40% in throughput and ↓26–32% in TTFT【measured, reports R2/R3】.

Q: What is the core prerequisite for this approach? A: The storage system must provide sufficiently low refill latency and high bandwidth (Mingxin measured cold-read bandwidth of 5.23 GB/s), the workload must exhibit a high prefix reuse rate, and the software stack must be deeply adapted (e.g., the LMCache parallel-read patch).

Q: How significant are the gains compared to the recomputation baseline? A: In Mingxin's measured R2 results, the no-external-storage recomputation baseline showed a TTFT p50 of 149.5s (concurrency 16), while FX100 achieved 11.85s—an acceleration of 8.6–20×; throughput increased from 4.1 to 74.9 tok/s【measured, report R2】.

References

1. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
2. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
3. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
4. NVIDIA CMX Context Memory Storage Platform — https://www.nvidia.com/en-us/data-center/ai-storage/cmx/