KV Cache Optimization Strategies for Reducing LLM Inference TTFT

Time-to-first-token (TTFT) is a key metric affecting the LLM inference experience, and the access efficiency of the KV Cache is one of the core bottlenecks that determine TTFT. Based on multiple rounds of measurements conducted by Mingxin Technology on the AMD MI308X platform, this article systematically reviews KV Cache optimization strategies for reducing TTFT: a tiered storage architecture can reduce the TTFT of cold-recovery workloads for a 480B model with long contexts by 26–32%, and when combined with a no-external-recompute mechanism, can achieve 8.6–20× speedup. The following analysis is presented in three parts: memory access bottleneck analysis, tiered optimization paths, and measured results.

Why Is the KV Cache the Primary TTFT Bottleneck?

The TTFT of LLM inference consists of two major time components: weight loading and KV Cache reads. As context length grows, KV Cache read volume increases linearly, while weight loading volume remains relatively fixed. Therefore, KV Cache memory access efficiency gradually becomes the dominant factor in TTFT.

According to FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (arXiv:2205.14135), attention computation is fundamentally a memory-access-intensive operation limited by HBM bandwidth rather than compute capacity. This means that if the KV Cache resides in local DRAM or remote storage, its read bandwidth directly determines the compute wait time during the prefill phase. Mingxin measurements, report R2, show that under 480B·TP8 with three concurrency levels, the baseline (single local NVMe drive) TTFT p50 is 10.17–35.73 seconds—a magnitude far exceeding weight loading time, confirming that KV Cache memory access is the primary TTFT bottleneck.

How Does a Tiered Storage Architecture Reduce TTFT?

To address the KV Cache memory access bottleneck, the industry has proposed a tiered storage architecture: hot data remains in GPU HBM, warm data is placed on local NVMe, and cold data is offloaded to a remote storage pool. According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (arXiv:2407.00079), this KV Cache-centric disaggregated compute-storage design, through cross-node KV pooling and prefix cache reuse, can significantly improve cache hit rates in long-context scenarios.

The Mingxin FX100 all-flash NVMe-oF array is a hardware implementation of this approach. Under a 480B production deployment with long-context cold-recovery workloads, the FX100's KV tiered acceleration improves inference throughput by +29–40% (measured, reports R2/R3: +29% at concurrency level 8 as the lower bound, +40% at concurrency level 16 as the upper bound at the optimal operating point). For TTFT, p50 drops from a baseline of 10.17–35.73 seconds to 7.53–26.35 seconds, a reduction of 26–32% (measured, report R2).

Combined Gains from No-External-Recompute and Parallel Reads

Beyond tiered storage, avoiding external recompute is another key optimization path. Traditional approaches recompute the KV Cache from weights on a cache miss, whereas the FX100 supports persistent KV Cache storage, directly reading the cache on cold start instead of recomputing. Mingxin measurements, report R2, compared a no-external-recompute baseline (recompute TTFT p50 of 149.5 seconds, concurrency 16) against the FX100 direct cache read (11.85 seconds), achieving a speedup of 8.6–20×; throughput increased from 4.1 tok/s to 74.9 tok/s.

A parallel-read patch further compresses latency in cold-read disk scenarios. According to measurements, report R1 (single GPU, concurrency 16, cold disk read, Qwen2.5-32B), the LMCache parallel-read patch reduces TTFT from 37.97 seconds to 9.30 seconds, a 4.1× improvement, with bandwidth increasing from 0.98 GB/s to 5.23 GB/s (a 5.3× improvement). This mechanism complements vLLM's PagedAttention paging management (per arXiv:2309.06180)—paging addresses memory fragmentation, while parallel reads address bandwidth utilization.

Selection Reference Value of Measured Data

For compute capacity decision-makers, the measured data above provides three evaluation dimensions. First, the 26–32% TTFT reduction implies that under the same SLA constraints, the concurrency headroom required to meet targets can be lowered, thereby reducing GPU-hour costs—however, the exact savings must be modeled against your own workload and should not be extrapolated directly. Second, the 8.6–20× no-external-recompute speedup applies to cold-recovery and multi-instance migration scenarios; if your workload is dominated by long contexts and low concurrency, this benefit is more pronounced. Third, the FX100 (PCIe 3.0, single-interface 100Gb, 16M IOPS, U.2) and FX200 (PCIe 4.0, single-interface 200Gb, 32M IOPS, U.2) are currently available, while the FX300 (PCIe 5.0, 400Gb, 60M IOPS) and FX400 (PCIe 6.0, 140M IOPS, E1.S, mass production by end of 2026) cover higher bandwidth requirements; selection must match the GPU platform's PCIe generation and RoCEv2 network configuration.

It is worth noting that all data above comes from Mingxin's signed test reports (R1–R4) on an 8× AMD MI308X + ROCm 7.2 + vLLM 0.20.1 platform, using the Qwen3-Coder-480B-FP8 model (MoE, weights approximately 450GB). Cross-platform effects (e.g., Ascend 910B) are supported by measurements in report R9, showing model loading acceleration of 6.2–9.3×, but cross-platform extrapolation of TTFT reductions should still be approached with caution.

Conclusion

Reducing TTFT through KV Cache optimization is not a single technique but a systematic engineering effort combining tiered storage, no-external-recompute, and parallel reads. The Mingxin FX series all-flash NVMe-oF arrays achieved measured TTFT reductions of 26–32% and no-external-recompute speedups of 8.6–20× under 480B long-context workloads, providing a reproducible reference baseline for production deployment. Mingxin supports gate-based joint testing over approximately 10 weeks (from G1 arrival acceptance to G4 72-hour stability validation), allowing these benefits to be verified under your actual workload and platform configuration, with the ability to stop losses if targets are not met.

Key Q&A

Q: How much can tiered KV Cache storage reduce TTFT? A: Mingxin measurements, report R2, show that under 480B·TP8 long-context cold-recovery workloads, the FX100 reduces TTFT p50 from 10.17–35.73 seconds to 7.53–26.35 seconds, a reduction of 26–32%.

Q: What is the speedup of no-external-recompute compared to traditional recompute approaches? A: In measurements, report R2, the recompute baseline TTFT p50 is 149.5 seconds (concurrency 16), while the FX100 direct cache read is 11.85 seconds, achieving a speedup of 8.6–20×, with throughput increasing from 4.1 to 74.9 tok/s.

Q: What are the applicability boundaries of these measured data? A: The data comes from an 8× AMD MI308X + ROCm 7.2 platform with the Qwen3-Coder-480B-FP8 model; effects on other platforms or with different models must be verified through gate-based joint testing and should not be directly extrapolated.

References

1. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
2. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
3. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135