Cross-Instance KV Cache Hot Sharing: Inference Acceleration in Multi-Service Deployments on a Single Machine
In single-machine multi-service instance inference deployments, cross-instance reuse of KV Cache has long been constrained by storage-layer bandwidth limitations and the
In single-machine multi-service instance inference deployments, cross-instance reuse of KV Cache has long been constrained by storage-layer bandwidth limitations and the absence of sharing mechanisms. In measured tests on the Mingxin FX100 with a 480B model and TP8 across three concurrency levels, cross-instance KV hot sharing via an NVMe-oF shared pool reduced time-to-first-token (TTFT) p50 from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32% [measured, report R2]. These results validate the feasibility of using a shared storage pool as a KV Cache carrier, providing a quantifiable technical path for GPU memory and compute reuse in multi-service deployments.
1. Why Does KV Cache Need Cross-Instance Sharing?
In large-model inference, KV Cache grows linearly with sequence length, with a single long-context request's KV data reaching several to tens of gigabytes. In a typical single-machine multi-service deployment (e.g., multiple vLLM instances on one physical machine, each serving different workloads), each instance maintains its own KV Cache independently, leading to two problems:
- GPU memory fragmentation: KV Cache memory reserved by each instance cannot be dynamically reallocated; long-context requests are prone to OOM, while KV memory in short-request instances sits idle.
- Redundant computation: When multiple instances process the same prefix (e.g., system prompts, shared documents), each recomputes KV independently, wasting compute and memory bandwidth.
The shared pool approach offloads KV Cache to an NVMe-oF storage array, enabling cross-instance KV reads and reuse through an intermediate layer such as LMCache. Measured data from the Mingxin FX100 shows that with a 480B model and TP8 configuration, TTFT reduction reaches 26% (lower bound) at concurrency level 8, and throughput improves by 40% (upper bound) at concurrency level 16 [measured, reports R2/R3]. This effect is particularly pronounced in long-context cold-restore workloads—where KV Cache must be rebuilt from storage, the shared pool's bandwidth advantage translates directly into latency gains.
2. How Does the Shared Pool Enable Cross-Instance KV Reuse?
The Mingxin FX100 shared pool implementation relies on the NVMe-oF (NVMe over Fabrics) protocol, mounting an all-flash array as a block device shared across multiple instances over a RoCEv2 network. At the software layer, LMCache serves as the KV cache management middleware, handling KV block writes, indexing, and cross-instance reads. Its workflow can be broken down as follows:
- Write path: After each vLLM instance generates KV, LMCache asynchronously writes it to the shared pool, building an index based on prefix hashes;
- Read path: When a new request arrives, LMCache first checks whether KV blocks matching the prefix exist in the shared pool; if a hit occurs, it reads directly; otherwise, it recomputes.
In measured tests, the LMCache parallel-read patch reduced TTFT from 37.97s to 9.30s in a single-GPU, concurrency-16, cold-read scenario (Qwen2.5-32B), a 4.1x improvement, with bandwidth increasing from 0.98 GB/s to 5.23 GB/s (↑5.3x) [measured, report R1]. This data reveals two implications: first, the shared pool's raw bandwidth (5.23 GB/s) is sufficient to support KV reads; second, LMCache's indexing and parallel-read mechanism is the key to reuse efficiency—if index lookups or read scheduling introduce bottlenecks, bandwidth advantages cannot translate into TTFT gains.
3. Performance Boundaries and Deployment Recommendations for Single-Machine Multi-Service Deployments
The shared pool does not accelerate unconditionally. Mingxin measured data shows a clear concurrency-dependent pattern: at concurrency level 8, KV-tiered acceleration improves inference throughput by 29% (lower bound), reaching 40% at concurrency level 16 (upper bound), with a whole-machine figure of 35–36% under TP4×2 [measured, report R3]. This indicates that reuse benefits scale with concurrent request count—the higher the concurrency, the greater the prefix overlap probability and the higher the KV hit rate.
Deployment recommendations are based on the following three points:
- Concurrency threshold: If single-instance concurrency is consistently below 8, the TTFT benefit of the shared pool (26–32%) still holds, but throughput improvement may fall below 30%. It is recommended to enable cross-instance sharing for production workloads with concurrency ≥16.
- Storage network: The shared pool requires a RoCEv2 lossless network; measured tests confirm bandwidth adequacy at 100GbE per port. If TCP/IP networking is used, latency will significantly impact the KV read path; RDMA deployment is strongly recommended.
- Cache policy: LMCache's indexing granularity directly affects hit rate. It is recommended to pre-populate KV caches based on business prefixes (e.g., system prompts, knowledge-base documents) rather than relying on runtime generation.
A notable boundary scenario is training checkpoint saving. The shared pool achieved a 178s→94s saving acceleration (1.9x, sustained write bandwidth 3.26→6.40 GB/s) in 8-GPU 32B LoRA training [measured, report R1]. This means the shared pool can serve both inference and training workloads, but the trade-off between write amplification and read latency must be considered—training writes consume bandwidth and may affect the real-time performance of inference KV reads.
4. Quantitative Comparison Against a Local NVMe Baseline
To clarify the net benefit of the shared pool, Mingxin tests used a local NVMe single drive (PCIe Gen4, 2TB) as the baseline. In scenarios without external-memory recomputation, the FX100 shared pool achieves 8.6–20x speedup over the recomputation baseline: the recomputation baseline TTFT p50 is 149.5s (concurrency 16), while the FX100 achieves 11.85s; throughput improves from 4.1 tok/s to 74.9 tok/s [measured, report R2]. This comparison demonstrates that for long-context cold-restore workloads, the shared pool's core value lies in eliminating recomputation—local NVMe, despite lower latency, cannot enable cross-instance KV reuse and forces each instance to recompute independently.
Key Q&A
Q: Under what conditions does cross-instance KV sharing yield the greatest benefit? A: With concurrent requests ≥16 and prefix overlap, throughput improvement can reach 40% (upper bound) [measured, report R3]; at concurrency level 8, the benefit drops to the 29% lower bound. Production workloads should prioritize ensuring concurrency scale.
Q: What is the core advantage of the shared pool over local NVMe? A: Eliminating cross-instance redundant computation. In scenarios without external-memory recomputation, the FX100 shared pool achieves 8.6–20x speedup over the recomputation baseline [measured, report R2]; local NVMe, while offering lower latency, cannot enable KV reuse across instances.
Q: What prerequisites are required for deploying a shared pool? A: A RoCEv2 lossless network (100GbE per port is sufficient), the LMCache middleware layer (parallel-read patch version), and concurrency loads of ≥16. When co-locating training and inference, the impact of write bandwidth on KV reads must be evaluated.
The Mingxin FX100 shared pool solution has completed sign-off-level test validation on the AMD MI308X platform, with collaboration adopting an approximately 10-week gate-based joint testing model (G1 arrival acceptance / G2 single-machine baseline / G3 main gate / G4 stability). Test models are Python-reproducible after NDA. For teams planning single-machine multi-service inference architectures, it is recommended to use the data in this article as a baseline and conduct small-scale validation tailored to their own workload characteristics.