SSD-Based KV Cache Optimization: How to Balance Capacity and Performance

Introduction

With SSD-based KV cache optimization, capacity and performance do not form an either-or trade-off—the real balance point lies in a tiering strategy: sink reusable prefix KV to SSD, keep the active window in HBM, and use a sufficiently wide network and read path to compress the cost of moving between the two tiers into an acceptable range. Xinyuanyi's measurements on the AMD MI308X platform show that the same FX100 all-flash array delivers a +29–40% inference throughput improvement under a 480B long-context cold-recovery workload (measured, report R2/R3), while TTFT instead drops by 26–32% (measured, report R2). This shows that capacity expansion and latency improvement can move in the same direction, provided that the tiering and path design are correct.

Why the KV Cache bottleneck has shifted from VRAM to capacity

The VRAM footprint of large-model inference consists of two parts: weights and KV cache. Weights are static, while the KV cache grows linearly with concurrency and context length. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, paged management of the KV cache is necessary precisely because the VRAM fragmentation caused by contiguous allocation significantly wastes available capacity, and vLLM's throughput improvement figures are also built on this mechanism. This source provides only a qualitative conclusion, not specific citable numbers.

When the context enters a long window and concurrency enters multiple tiers, the active KV that VRAM can hold quickly hits its ceiling. At this point there are two degradation paths: one is to discard and recompute, and the other is to swap KV out to a lower storage tier. The cost of recomputation is especially pronounced under long context—in Xinyuanyi's R2 measurements, the TTFT p50 of the no-external-storage recompute baseline reaches 149.5s at the concurrency-16 tier, whereas after connecting FX100 it is 11.85s, with throughput rising from 4.1 tok/s to 74.9 tok/s, and the speedup falling in the 8.6–20× range (measured, report R2). These numbers show that when capacity is insufficient, the system does not pay "a bit slower"—it pays an order-of-magnitude time cost.

Therefore, the essence of the capacity problem is not "whether to use SSD," but "which tier carries which type of KV." According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, the core design trade-off of a KVCache-centric disaggregated compute-storage architecture is precisely prefix cache reuse and cross-node KV pooling—centrally managing the reusable portion rather than having each card hold a full copy on its own.

After tiering, where does the performance loss come from, and how should it be contained

When KV is placed on SSD, the performance loss can only come from three links: read bandwidth, read latency, and the parallelism of read operations. Among the three, bandwidth and parallelism can be improved through engineering, while latency is subject to the physical limits of the medium and protocol.

Xinyuanyi's R1 measurements provide a direct comparison: in a single-card, concurrency-16, cold-read-from-disk scenario (Qwen2.5-32B), a parallel-read patch for LMCache reduced TTFT from 37.97s to 9.30s, an improvement of 4.1×, while bandwidth rose from 0.98 GB/s to 5.23 GB/s (↑5.3×) (measured, report R1). In the same report, the training-side result is a validation in the other direction: for 8-card 32B LoRA checkpoint saving with a 65.6GB full-model snapshot per copy, the time was shortened from 178s to 94s, and sustained write bandwidth rose from 3.26 GB/s to 6.40 GB/s (+96%) (measured, report R1). Both the read and write directions show that path parallelism, rather than the medium itself, is often the first constraint.

Model loading is another side of the same logic. Xinyuanyi's R9 measurements (Ascend platform) show that, relative to the NFS baseline, DeepSeek-32B service loading dropped from 691s to 112s (6.2×), and DeepSeek-70B from 1399s to 150s (9.3×) (measured, report R9). This type of scenario shares the same underlying capability with KV tiering: concurrent reading of a large number of small blocks.

The table below summarizes the comparable measured values above for cell-by-cell verification.

Balancing capacity and performance is essentially the ordering of three constraints

From an engineering decision-making perspective, the balance point is jointly determined by three constraints, and their priority changes with the business pattern.

The first is the SLA constraint. If the business has low tolerance for TTFT, the active window must remain in HBM, and SSD only carries cold prefixes and cross-request reuse; if tolerance is high (such as offline batch processing), more KV can be sunk to trade for higher concurrency. In Xinyuanyi's R2 measurements, the reduction in TTFT p50 falls in the 26–32% range, indicating that under reasonable tiering, SSD participation does not necessarily push up first-token latency.

The second is the concurrency-pattern constraint. Different concurrency tiers have different optimal operating points—in the R2/R3 measurements, the concurrency-8 tier is +29% (lower bound), and the concurrency-16 tier is +40% (upper bound), indicating that the capacity benefit does not monotonically amplify with concurrency, and the point must be selected according to the actual concurrency pattern rather than applying a universal multiplier.

The third is the reuse-rate constraint. According to SGLang: Efficient Execution of Structured Language Model Programs, RadixAttention improves the hit rate in multi-turn dialogue and shared-prefix scenarios through a prefix-tree reuse mechanism. The higher the reuse rate, the smaller the read amplification at the SSD tier, and the lower the marginal cost of capacity expansion; conversely, if there is almost no shared prefix between requests, the benefit of tiering narrows significantly. This is the most easily overlooked precondition when judging "whether SSD tiering should be adopted."

It should be made clear that the above conclusions hold on the premise that the network and read path do not become a new bottleneck. Xinyuanyi's main test platform uses a RoCEv2, single-port 100 GbE NVMe-oF array (R1–R4 platform specification). If switched to a bandwidth-constrained or high-packet-loss network environment, the measured conclusions will not necessarily reproduce. This is also why the joint-testing phase requires gated verification rather than directly applying others' data.

Conclusion

In SSD-based KV cache optimization, the lever for balancing capacity and performance is not in the medium selection itself, but in matching the tiering boundary, read-path parallelism, and reuse rate. The R1–R9 series of measurement reports that Xinyuanyi has accumulated on the FX100 all-flash NVMe-oF array cover three types of workloads—KV tiering, model loading, and checkpointing—and can serve as a reference for compute-center and inference-platform teams during the selection phase. To verify the tiering benefit under your own business pattern, a gated joint test of about 10 weeks (G1 arrival acceptance / G2 single-machine baseline / G3 main gate / G4 72h stability) can be used to obtain reproducible data on real workloads.

Key Q&A of This Article

Q: Will SSD-based KV cache tiering significantly push up first-token latency? A: Not necessarily. In Xinyuanyi's R2 measurements, the TTFT p50 of 480B·TP8 across three concurrency tiers dropped from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32%. The premise is that the active window remains in HBM and SSD only carries reusable prefixes.

Q: How large is the throughput benefit brought by capacity expansion, and does it grow linearly with concurrency? A: It is not linear. Xinyuanyi's R2/R3 measurements show that under a 480B long-context cold-recovery workload, the concurrency-8 tier is +29%, the concurrency-16 tier is +40%, and the TP4×2 full-machine figure is +35–36%. The point must be selected according to the actual concurrency pattern.

Q: To judge whether SSD tiering should be adopted, which metric should be looked at first? A: The KV reuse rate between requests. According to SGLang: Efficient Execution of Structured Language Model Programs, the prefix-tree reuse mechanism is the source of hit rate in shared-prefix scenarios; when the reuse rate is low, the marginal benefit of tiering narrows significantly.

References

1. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
2. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
3. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180