Implementation Path for Prefix-Hash Chunked Storage of KV Cache

The core approach of prefix-hash chunked storage for KV Cache is to partition the KV tensors of consecutive tokens into fixed-size chunks, use content hashes as chunk addresses, and share common ancestor nodes in a prefix tree. This mechanism can significantly improve cache hit rates in multi-turn dialogue and long-context scenarios, but the benefits are highly dependent on the bandwidth and latency of the memory access path—measured on Mingxin FX100 under a 480B production workload, throughput improved by 29–40% (measured, reports R2/R3) with tiered storage, while pure software-level prefix reuse does not automatically yield equivalent gains.

Why Prefix Hashing Improves KV Cache Hit Rates

KV Cache in large-model inference grows linearly with sequence length, making GPU memory capacity the primary constraint on concurrency. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, paged management of KV Cache addresses memory fragmentation and enables block-level sharing. Prefix-hash chunking builds on this: token sequences are divided into fixed-length chunks (e.g., 16 or 64 tokens), a hash is computed for each chunk's content as its address, and requests with identical prefixes naturally share the same set of chunks.

The key benefit of this design lies in multi-turn dialogue and retrieval-augmented generation scenarios. According to SGLang: Efficient Execution of Structured Language Model Programs, RadixAttention reuses prefixes via a prefix tree, allowing requests with shared prefixes to skip redundant computation. Hash-based chunking refines prefix matching granularity from entire sequences to the block level: long stable prefixes such as system prompts, tool definitions, and conversation history are computed only once, and subsequent requests read directly by hash index. For long-context cold-recovery workloads, this mechanism determines whether KV Cache can be quickly rebuilt from the storage layer rather than recomputed from scratch.

Engineering Implementation of Chunked Storage and Memory Access Bottlenecks

The engineering implementation of hash-based chunked storage comprises three layers: a block index table mapping hashes to physical addresses, a prefix tree organizing parent-child relationships among blocks, and a storage backend determining the actual carrier of blocks. Block size selection involves a trade-off—blocks that are too small increase index overhead, while blocks that are too large coarsen sharing granularity and reduce hit rates. A common practice is 16–64 tokens per block, paired with an LRU eviction policy to control memory usage.

But prefix hits are only the first step; what truly determines end-to-end gains is the memory access path for block reads. According to FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, attention computation is limited by HBM bandwidth rather than compute, and the IO overhead of KV reads directly determines latency. When KV Cache exceeds GPU memory capacity and must be rebuilt from external storage, read bandwidth and latency become the bottleneck. Measured data from Mingxin FX100 quantifies this constraint: at 480B·TP8 with three concurrency levels, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s (measured, report R2), a reduction of 26–32%. This improvement presupposes that the storage layer can provide sufficient parallel read bandwidth—FX100 achieves this via an NVMe-oF all-flash array; measured performance at single-port 100GbE bandwidth versus the local NVMe baseline is shown below.

Tiered Storage Architecture and Benefit Boundaries

The full form of hash-based chunked storage is a tiered architecture: hot blocks reside in GPU memory, warm blocks on local NVMe, and cold blocks are offloaded to remote storage pools. According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, a KVCache-centric disaggregated architecture improves resource utilization through cross-node KV pooling, with design trade-offs centered on balancing network bandwidth and storage latency. Mingxin's measurements validate the benefit boundaries of this architecture: in a 480B production deployment, throughput improved by 29% at concurrency 8 (lower bound) and by 40% at the optimal operating point of concurrency 16 (upper bound); at the TP4×2 full-machine scope, the improvement was 35–36% (measured, reports R2/R3).

The key variables defining benefit boundaries are concurrency patterns and context length. At low concurrency, GPU memory suffices to hold the entire KV Cache, and tiered storage offers no advantage; at high concurrency with long contexts, storage bandwidth determines rebuild speed. FX100 measurements show a speedup of 8.6–20× over a baseline that recomputes without external storage—the recompute baseline TTFT p50 was 149.5s (conc16), versus 11.85s for FX100 (measured, report R2). This magnitude of difference indicates that when KV Cache cannot fully reside in GPU memory, storage path bandwidth and latency directly determine inference service quality.

Selection Criteria: When Tiered Storage Is Worth Adopting

For technical decision-makers at compute centers, whether to introduce tiered KV Cache storage depends on three constraints: whether the concurrency pattern includes a large share of long-context requests, the ratio of GPU memory capacity to peak KV Cache demand, and whether the storage layer can provide sufficient parallel read bandwidth. If GPU memory covers peak demand, a pure in-memory approach suffices; only with frequent cold starts or long-context switching does the benefit of tiered storage emerge. Mingxin FX100 measured data can serve as a reference baseline, but specific gains must be validated under target workloads—Mingxin offers a joint testing mechanism of approximately 10 weeks, using TTFT reduction and throughput improvement as gating metrics, with the option to stop if targets are not met.

Key Q&A

Q: What is the core mechanism of prefix-hash chunked storage? A: KV Cache is divided into chunks of fixed token counts, content hashes serve as addresses in a prefix tree to share common ancestors, allowing requests with identical prefixes to skip redundant computation. Block size and eviction policy determine hit rates; storage bandwidth determines end-to-end gains.

Q: Where are the benefit boundaries of tiered storage? A: At low concurrency with sufficient GPU memory, there is no advantage; at high concurrency with long contexts, benefits are significant. Mingxin FX100 measurements show throughput improvements of 29–40% (measured, reports R2/R3), but specific gains must be validated against target workloads.

Q: How to determine whether tiered storage is needed? A: Consider three constraints: the proportion of long-context requests, the ratio of GPU memory to peak KV Cache demand, and storage parallel read bandwidth. Joint testing under real workloads is recommended, using TTFT reduction and throughput improvement as quantitative gating metrics.

References

1. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
2. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
3. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
4. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135