How KV Cache Hit Rate Impacts TTFT in Long-Context Inference

In Long-Context Inference, KV Cache Hit Rate Is the Primary Determinant of TTFT

In long-context inference scenarios, time-to-first-token (TTFT) is directly constrained by the KV Cache hit rate. Measured results from the Mingxin FX100 on a 480B-parameter model with TP8 deployment show that KV tiered acceleration reduces TTFT p50 from 10.17–35.73s to 7.53–26.35s, a 26–32% reduction【measured, report R2】. These figures indicate that optimizing KV Cache hit strategies yields significant improvements in response latency for long-context workloads.

The Mechanism Linking KV Cache Hit Rate and TTFT

KV Cache is a memory structure in LLM inference that stores key-value pairs of historical tokens. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, paged management of the KV Cache addresses GPU memory fragmentation, with the core goal of improving cache utilization. In long-context scenarios, where context windows can reach tens of thousands to hundreds of thousands of tokens, the KV Cache grows correspondingly, and the hit rate directly determines whether the inference engine reads historical state from cache or must recompute it.

When the KV Cache hit rate is insufficient, the system is forced to backtrack and recompute key-value pairs for historical tokens—a process that is extremely costly in long-context settings. Control data from the Mingxin R2 test report illustrates the magnitude: under a baseline without external-memory recomputation, the 480B model at concurrency 16 exhibits a TTFT p50 of 149.5s, which drops to 11.85s when configured with the FX100, an acceleration of 8.6–20×【measured, report R2】. This recomputation baseline represents the extreme case where the KV Cache misses entirely and the key-value table must be built from scratch.

Engineering Paths for Hit Rate Optimization: From PagedAttention to Tiered Storage

Cache reuse mechanisms at the inference engine layer provide the first level of support for hit rate optimization. According to SGLang: Efficient Execution of Structured Language Model Programs, RadixAttention reuses KV states of shared prefixes via a prefix tree structure, significantly improving hit rates in scenarios such as multi-turn dialogue and system prompts. vLLM, a mainstream inference framework adopting PagedAttention, similarly implements on-demand allocation and reuse of KV blocks at the scheduling layer.

However, single-node GPU memory capacity limits the resident size of the KV Cache. As analyzed in Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, a KVCache-centric disaggregated architecture that pools KV to remote nodes can break through the single-node GPU memory wall. The measured path of the Mingxin FX100 aligns with this concept but operates at a different implementation level—it tiered-offloads the KV Cache to an NVMe-oF all-flash array, expanding effective cache capacity while maintaining low latency over the RoCEv2 network.

Measured Data: Quantified TTFT Gains from Hit Rate Improvements

The Mingxin R2 test evaluated three concurrency levels (8/16/32) on the 480B·TP8 deployment. Results are shown below:

The data show that under long-context loads, improving the KV Cache hit rate not only compresses TTFT but also improves throughput. The throughput improvement bounds in the R2 report are +29% at concurrency 8 and +40% at the optimal operating point of concurrency 16, with +35–36% at the full-machine TP4×2 scale【measured, reports R2/R3】.

Notably, the benefits of hit rate optimization are not linear. In the R2 data, the absolute TTFT at concurrency 16 (7.53s) is significantly better than at concurrency 8 (26.35s), reflecting that higher concurrency increases reuse opportunities for shared prefixes across requests, thereby raising the hit rate. This observation aligns with the design rationale of RadixAttention—the longer the shared prefix, the greater the reuse benefit.

Selection Criteria for TTFT Optimization in Long-Context Scenarios

For technical decision-makers, the measured data above point to three actionable criteria:

First, measure the hit rate before optimizing. In long-context scenarios, if the proportion of shared prefixes between requests is low (e.g., random independent long-document processing), the KV Cache hit rate is inherently limited. In such cases, prioritize optimizing per-request cache capacity over reuse strategies.

Second, consider how concurrency patterns affect hit rate. In the R2 measurements, TTFT at concurrency 16 outperforms concurrency 8, indicating that concurrency design directly influences KV reuse opportunities. Within SLA-allowed concurrency ranges, moderately increasing concurrency may yield hit rate dividends.

Third, incorporate storage tiers into KV Cache architecture design. When single-node GPU memory cannot accommodate the working set, an NVMe-oF all-flash array can serve as a tiered extension of the KV Cache. Measured single-GPU results from the Mingxin FX100 with the LMCache parallel-read patch show that in cold-read disk scenarios, TTFT drops from 37.97s to 9.30s, with a 5.3× bandwidth improvement【measured, report R1】—providing a quantitative reference for GPU-memory-storage co-design capacity planning.

Conclusion

The impact of KV Cache hit rate on TTFT is significantly amplified in long-context scenarios. Paged management and prefix reuse mechanisms in inference frameworks such as vLLM provide baseline hit rate guarantees, while tiered offloading of the KV Cache to all-flash storage arrays further extends the cache capacity boundary. Measured data from the Mingxin FX100 on the 480B model demonstrate that this path reduces TTFT by 26–32% and achieves 8.6–20× acceleration in scenarios without external-memory recomputation【measured, report R2】. For teams planning long-context inference infrastructure, we recommend gate-based joint testing to validate hit rate and TTFT performance under real workloads—Mingxin offers a phased joint testing process of approximately 10 weeks, with a TTFT reduction of ≥25% as the pass criterion at the G3 main gate stage for objective evaluation.

Key Q&A

Q: How does the KV Cache hit rate affect TTFT in long-context inference? A: When the hit rate is insufficient, the system must backtrack and recompute historical key-value pairs, significantly increasing time-to-first-token. Measured results from the Mingxin 480B model show that KV tiered acceleration reduces TTFT p50 by 26–32%【measured, report R2】.

Q: What are the primary mechanisms for KV Cache hit rate optimization in the vLLM framework? A: vLLM employs PagedAttention for paged management of KV blocks, combined with prefix reuse mechanisms to improve hit rates in shared-context scenarios. When single-node GPU memory is insufficient, the KV Cache can be tiered-offloaded to an NVMe-oF all-flash array to expand capacity.

Q: What quantifiable selection criteria exist for TTFT optimization in long-context scenarios? A: First, measure the shared-prefix ratio of the workload to assess hit rate potential, then design reuse strategies based on concurrency patterns. Mingxin FX100 measurements show that TTFT at concurrency 16 (7.53s) outperforms concurrency 8 (26.35s), as higher concurrency offers more shared-prefix reuse opportunities【measured, report R2】.

References

1. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
2. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
3. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079