Local LLM Deployment: KV Cache Offloading to Memory and the Boundaries of VRAM Residency

In local LLM deployment, offloading KV Cache to memory versus keeping it resident in VRAM each have clearly defined applicable boundaries; there is no universally optimal solution. The core criteria are concurrency tier and context length: in low-concurrency, short-context scenarios, VRAM residency is advantageous; in high-concurrency, long-context scenarios, memory offloading yields measurable gains through tiered acceleration. Under a 480B production-form long-context cold-recovery workload, the Xinyuanyi FX100 measured a throughput improvement of +29–40% (measured, report R2/R3), which can serve as a reference for the effectiveness of the memory-offloading path.

Why KV Cache Becomes a Deployment Bottleneck

During the Transformer decoding phase, each token must read the entire historical KV Cache, and memory access density grows linearly with sequence length. According to "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness," attention computation is limited by HBM bandwidth rather than compute, and this memory-access bottleneck fundamentally determines that the storage location of KV Cache directly affects decoding throughput.

VRAM capacity constraints make the problem more explicit. According to "Efficient Memory Management for Large Language Model Serving with PagedAttention," paged KV Cache management solves the VRAM fragmentation problem but does not raise the available capacity ceiling. When context length or concurrency approaches the VRAM boundary, the system is forced to trade off between swap-out recomputation and capacity planning.

The typical contradiction in local deployment scenarios is: the total VRAM of a single card or multiple cards is fixed, while long context and multi-user concurrency simultaneously amplify KV Cache demand. At this point, offloading KV Cache to memory becomes a structural solution—memory capacity is far larger than VRAM, and in local deployment scenarios memory resides on the same machine as the GPU, making the access path controllable.

Where the Gains from Memory Offloading End

Memory offloading is not unconditionally superior to VRAM residency. Its gain premise is that KV Cache access exhibits temporal locality, and recently used prefixes are read repeatedly. According to "SGLang: Efficient Execution of Structured Language Model Programs," RadixAttention improves hit rates in shared-prefix scenarios through prefix-tree reuse, and this mechanism applies equally to tiered storage designs.

Measured data from Xinyuanyi under a 480B·TP8 long-context cold-recovery workload can serve as a reference. Across three concurrency tiers, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32% (measured, report R2); throughput improvement was +29% at concurrency 8 and +40% at concurrency 16 (measured, report R2/R3). These data indicate that the higher the concurrency and the longer the context, the more significant the gains from memory offloading.

In contrast, in low-concurrency, short-context scenarios, the total KV Cache is small, VRAM is sufficient to hold the entire cache, and the PCIe or network round trips introduced by memory offloading instead add latency. In this case, VRAM residency is the more direct choice—the access path is shortest, with no cross-device copy overhead.

Cold Recovery and Recomputation Are the Key Divide

The core variable distinguishing the applicability of the two paths is cache hit rate. In high-hit-rate scenarios (multi-turn dialogue, shared prefixes, batched requests with identical prefixes), the read cost of memory offloading is amortized by hit gains; in low-hit-rate scenarios (random long documents, cold start), every miss in external reads incurs the full memory-access cost.

According to "Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving," a KV Cache-centric disaggregated architecture improves resource utilization through cross-node KV pooling, and its design premise is the high hit rate brought by prefix cache reuse. This architectural logic holds equally in single-machine tiered scenarios: hit rate determines offloading gains.

A noteworthy comparison in Xinyuanyi's measurements is the "recomputation without external storage" baseline: the recomputation baseline TTFT p50 was 149.5s (concurrency 16) versus 11.85s for the FX100, with throughput of 4.1 versus 74.9 tok/s (measured, report R2). This extreme comparison shows that when the cache does not fit and the system falls back to recomputation, the cost far exceeds that of any tiering strategy. Therefore, the first principle of selection is to avoid recomputation; the second principle is to choose the offloading tier.

Selection Criteria and Validation Path

Based on the above analysis, the storage location of KV Cache in local deployment can be decided in the following order:

1. VRAM headroom accounting: Estimate the total KV Cache under the target concurrency and context length. If it is below the safety threshold of available VRAM capacity (it is recommended to evaluate after reserving VRAM for weights and activations), prioritize VRAM residency.
2. Hit rate assessment: If the workload is characterized by multi-turn dialogue or shared prefixes, the gain potential from memory offloading is larger; if it consists of random short requests, offloading gains are limited.
3. Dedicated cold-recovery testing: Long-context cold recovery is the worst case; if this scenario meets the bar, regular scenarios are safe.

The Xinyuanyi FX100 completed the above validation on an AMD MI308X×8 platform (ROCm 7.2, vLLM 0.20.1+rocm721). The G3 main gate uses a TTFT reduction of ≥25% and a throughput of +29–40% within the measured band as the pass criteria (measured, report R2/R3). This test environment and its conclusions apply to comparable ROCm platforms; migration results on CUDA platforms require separate validation.

Xinyuanyi offers an approximately 10-week gated joint test (G1 arrival acceptance through G4 72h stability), supporting validation of tiering gains on your own workload, with a stop-loss if the bar is not met. To evaluate the KV Cache tiering effect for a specific model, it can be measured directly during the joint test.

Key Questions and Answers for This Article

Q: How do you decide whether KV Cache goes in memory or VRAM? A: First account for the total KV Cache under the target concurrency and context. If VRAM can hold it, prioritize residency; if capacity is insufficient and the workload has prefix-reuse characteristics, memory offloading can yield gains through tiered acceleration.

Q: How large are the gains from memory offloading? A: Under a 480B long-context cold-recovery workload, the Xinyuanyi FX100 measured a throughput improvement of +29–40% (measured, report R2/R3) and a TTFT p50 reduction of 26–32% (measured, report R2), with gains increasing with concurrency and context length.

Q: What is the prerequisite for memory offloading to apply? A: Cache hit rate is the core prerequisite. High-hit-rate scenarios (multi-turn dialogue, shared prefixes) show significant gains; low-hit-rate scenarios show limited offloading gains, and falling back to recomputation when the cache does not fit should be avoided—the recomputation baseline TTFT p50 is as high as 149.5s (measured, report R2).

References

1. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
2. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
3. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
4. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079