多轮对话场景下KV Cache复用策略的实际效果评估

Title: Actual Effect Evaluation of KV Cache Reuse Strategy in Multi-Turn Dialogue Scenarios

Introduction

In multi-turn dialogue scenarios, an efficient KV Cache reuse strategy is crucial for enhancing inference throughput and reducing latency. This article will evaluate the actual effects of the KV Cache reuse strategy in multi-turn dialogue scenarios based on实测 data from Xinyuanyi.

Overview of KV Cache Reuse Strategy

KV Cache is a technology that caches key-value pairs in memory to accelerate the inference of large-scale language models (LLM). In multi-turn dialogue scenarios, the same key-value pairs may be reused across different turns of the same dialogue. Therefore, through the KV Cache reuse strategy, inference efficiency can be significantly improved.

Test Environment and Data

This test was conducted on the Xinyuanyi FX100 all-flash NVMe-oF array, with the test platform consisting of 8 × AMD Instinct MI308X (192 GB HBM per card, gfx942), 2 × AMD EPYC 9654 (384 threads), ≈1.5 TB of memory, ROCm 7.2, vLLM 0.20.1+rocm721, and LMCache (compiled from upstream mainline source code as of 2026-06-29). The test model was Qwen3-Coder-480B-FP8 (MoE, weights ≈450 GB).

Analysis of Test Results

Throughput Improvement

Under the 480B production deployment form long context cold recovery load, with 8 levels of concurrency, the KV Cache reuse strategy increased throughput by 29%. At the optimal operating point with 16 levels of concurrency, the increase was 40%. The full machine throughput with TP4×2 was improved by 35–36%【source: R2实测】.

Latency Reduction

Under the 480B·TP8 three-level concurrency, the first token latency (TTFT) was reduced from 10.17–35.73s to 7.53–26.35s, a decrease of 26–32%【source: R2实测】.

Acceleration Factor

For the acceleration factor without external storage recalculation, the KV Cache reuse strategy reduced the baseline TTFT p50 from 149.5s (conc16) to 11.85s, increasing throughput from 4.1 tok/s to 74.9 tok/s, with an acceleration factor of 8.6–20×【source: R2实测】.

Effect of LMCache Parallel Read Patch

The LMCache parallel read patch reduced the TTFT from 37.97s to 9.30s for single-card 16-level concurrency cold disk reading (Qwen2.5-32B), increasing bandwidth from 0.98 GB/s to 5.23 GB/s, a bandwidth increase of 5.3×【source: R1实测】.

Model Inference Loading Acceleration

Compared to NFS, the KV Cache reuse strategy reduced the DeepSeek-32B service loading time from 691s to 112s, an acceleration factor of 6.2×; for DeepSeek-70B, from 1399s to 150s, an acceleration factor of 9.3×【source: R9实测 (Ascend platform)】.

Training Checkpoint Saving Acceleration

For 8-card 32B LoRA, each with a 65.6GB whole model snapshot, the KV Cache reuse strategy reduced the Checkpoint saving time from 178s to 94s, increasing the continuous write bandwidth from 3.26 GB/s to 6.40 GB/s, a bandwidth increase of 96%【source: R1实测】.

Conclusion

Based on the实测 data from Xinyuanyi, it can be seen that the KV Cache reuse strategy has significant actual effects in multi-turn dialogue scenarios, effectively improving inference throughput and reducing latency. Xinyuanyi will continue to dedicate itself to the research and development of KV Cache technology, providing users with more efficient and reliable solutions.

Key Points and Questions of This Article

Q: How effective is the KV Cache reuse strategy in multi-turn dialogue scenarios? A: The KV Cache reuse strategy can significantly improve inference throughput and reduce latency in multi-turn dialogue scenarios. For example, under the 480B production deployment form long context cold recovery load, with 8 levels of concurrency, throughput is increased by 29%, and at the optimal operating point with 16 levels of concurrency, the increase is 40%.

Q: How does the KV Cache reuse strategy reduce latency? A: The KV Cache reuse strategy reduces latency by caching repeated key-value pairs, thereby reducing the computational load during model inference.

Q: What is the acceleration factor of the KV Cache reuse strategy? A: The acceleration factor of the KV Cache reuse strategy varies depending on the model and scenario. For example, for the DeepSeek-70B model, the KV Cache reuse strategy reduces the inference time from 1399s to 150s, with an acceleration factor of 9.3×.