Why vLLM's KV Cache Allocation Strategy Determines Throughput, Not Just VRAM Usage
The impact of KV Cache allocation strategy on vLLM throughput goes far beyond the question of "is there enough VRAM." The core mechanism is this
The impact of KV Cache allocation strategy on vLLM throughput goes far beyond the question of "is there enough VRAM." The core mechanism is this: the allocation strategy determines the degree of VRAM fragmentation and the total available KV Cache capacity, which in turn directly constrains the inference engine's per-batch concurrency (batch size) and scheduling efficiency—and these two factors are what ultimately determine throughput. Measured on Mingxin FX100 under a 480B model long-context workload, optimizing KV tiered storage and memory access paths improved inference throughput by 29–40% (measured, reports R2/R3), confirming the significant leverage that allocation and memory access path optimization exert on throughput.
How VRAM Allocation Propagates to Throughput: From Fragmentation to Batch Scheduling
vLLM's core innovation lies in the PagedAttention mechanism, which splits KV Cache from contiguous VRAM blocks into fixed-size pages, thereby eliminating VRAM fragmentation and enabling near-zero-waste memory sharing (per "Efficient Memory Management for Large Language Model Serving with PagedAttention"). But the impact of the allocation strategy does not stop at VRAM utilization—it directly determines how many concurrent sequences the inference engine can accommodate in a single decoding step.
In autoregressive decoding, throughput scales approximately linearly with batch size: the larger the batch, the more saturated the GPU compute resources, and the more tokens produced per unit time. The upper bound on batch size depends on whether the available KV Cache capacity can hold the KV tensors of all active sequences. If the allocation strategy causes fragmentation or insufficient reservation, the engine is forced to shrink the batch size, GPU compute cycles go idle, and throughput drops.
A notable observation from Mingxin's R2 measurements: under a 480B·TP8 long-context cold-restore workload, throughput gains showed a clear concurrency-dependent pattern—29% improvement at concurrency level 8 (lower bound), rising to 40% at the optimal operating point of concurrency 16 (upper bound) (measured, reports R2/R3). This nonlinear relationship demonstrates precisely that as concurrency increases, KV Cache allocation and memory access efficiency act as a bottleneck magnifier, and the benefits of optimizing the allocation strategy grow accordingly.
The Link Between Time-to-First-Token and Throughput: An Indirect Channel for Allocation Strategy
KV Cache allocation strategy also indirectly affects throughput through time-to-first-token (TTFT). In long-context cold-start scenarios, the prefill phase must read historical KV from external storage into VRAM. If the allocation strategy fails to reserve sufficient contiguous space for prefill, or if the memory access path is not optimized, TTFT degrades sharply—and TTFT is a core service-level agreement (SLA) metric.
Mingxin's R2 measurements show that under a 480B·TP8 workload at three concurrency levels, TTFT p50 dropped from a baseline (local single NVMe drive) of 10.17–35.73 seconds to 7.53–26.35 seconds, a reduction of 26–32% (measured, report R2). Lower TTFT means that under the same SLA constraints, the system can sustain higher concurrency pressure without violating latency commitments—which is equivalent to increasing effective throughput capacity without adding hardware.
This logic is even clearer in the extreme scenario of no external recomputation: baseline TTFT p50 with recomputation reached 149.5 seconds (concurrency 16), while FX100 required only 11.85 seconds, with throughput rising from 4.1 to 74.9 token/s (measured, report R2). When KV does not need recomputation, optimizing the allocation strategy directly eliminates compute waste, and its throughput leverage far exceeds conventional scheduling optimizations.
Prefix Reuse and KV Pooling: Architectural Extensions of Allocation Strategy
Another throughput lever for allocation strategy lies in prefix reuse. The RadixAttention mechanism proposed by SGLang organizes KV Cache via a prefix tree, enabling direct reuse of KV for multi-turn conversations and shared document prefixes, avoiding redundant computation (per "SGLang: Efficient Execution of Structured Language Model Programs"). Mooncake goes further by decoupling KVCache from single-GPU VRAM, building a KVCache-centric disaggregated compute-storage architecture that enables cross-node KV pooling and scheduling (per "Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving").
Mingxin FX100's KV tiered acceleration approach shares the same architectural philosophy: by placing KV Cache in tiers across near-memory media (all-flash NVMe-oF arrays) rather than relying solely on VRAM, combined with an efficient parallel read path, quantifiable gains are achieved in cold-read scenarios. R1 measurements show that under a single-GPU, concurrency-16, cold-read workload (Qwen2.5-32B), the LMCache parallel-read patch reduced TTFT from 37.97 seconds to 9.30 seconds (4.1×), and bandwidth improved from 0.98 to 5.23 GB/s (measured, report R1). The essence of this improvement is that the allocation strategy expands from "static allocation within VRAM" to "dynamic scheduling across storage tiers," so that available KV capacity is no longer bounded by the physical VRAM limit of a single GPU.
Selection Criteria for Allocation Strategy: From Measurements to Decisions
For technical decision-makers at compute centers, selecting a KV Cache allocation strategy should focus on three dimensions. First, whether the engine layer adopts paged management to eliminate fragmentation (e.g., vLLM's PagedAttention). Second, whether prefix reuse is supported to reduce redundant computation (e.g., SGLang's RadixAttention). Third, whether KV tiered offloading capability exists, allowing KV capacity to break past the single-GPU VRAM boundary.
Mingxin FX100's measured data on the third dimension above can serve as a reference: on the Huawei Atlas 910B platform, model inference loading (including KV initialization) accelerated 6.2–9.3× compared to the NFS baseline—DeepSeek-32B service loading dropped from 691 seconds to 112 seconds, and DeepSeek-70B from 1399 seconds to 150 seconds (measured, report R9, Ascend platform). On the training side, Checkpoint saving also achieved 1.9× acceleration (178 s → 94 s, sustained write bandwidth 3.26 → 6.40 GB/s) (measured, report R1). These figures indicate that storage media memory access performance directly determines the feasibility boundary of KV tiering schemes.
Key Q&A from This Article
Q: Why does KV Cache allocation strategy affect throughput rather than just VRAM usage? A: The allocation strategy determines the degree of VRAM fragmentation and total available KV capacity, which directly constrains the upper bound of batch concurrency. The lower the concurrency, the more GPU compute cycles sit idle, and the lower the throughput. Mingxin's R2 measurements show that optimizing KV memory access paths under a 480B long-context workload yields 29–40% throughput improvement, with greater gains at higher concurrency.
Q: What is the linkage between TTFT and throughput? A: TTFT is a core SLA metric. The lower the TTFT, the higher the concurrency the system can sustain under the same latency constraints, and the greater the equivalent throughput capacity. In Mingxin's R2 measurements, TTFT p50 decreased by 26–32%, providing headroom for higher effective throughput under concurrency pressure.
Q: How is the throughput value of KV tiered offloading validated? A: KV tiering allows cache capacity to exceed the physical VRAM limit of a single GPU, but it depends on the memory access performance of the storage media. Mingxin's R1 measurements show a 4.1× TTFT improvement and 5.3× bandwidth improvement from the parallel-read patch in cold-read scenarios; R9 measurements on the Ascend platform show 6.2–9.3× acceleration in model loading, validating the ability of high-speed storage media to support KV tiering schemes.
References
- Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
- SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
- Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
Mingxin Technology has long been engaged in joint testing of KV Cache tiered acceleration with domestic compute platforms. If you wish to validate the actual impact of different KV allocation strategies on throughput in your own cluster, reproducible quantitative conclusions can be obtained through our gated joint-testing process (approximately 10 weeks, including in-band measurements of TTFT reduction and throughput improvement).