Does KV Cache Memory Grow Linearly When Context Doubles?

KV Cache memory grows with context length—strictly linear at the formula level—but deployments often show apparent super-linear growth. This is not the cache itself being super-linear, but the combined result of management overhead, fragmentation, and recomputation strategies. Understanding this distinction is a prerequisite for estimating long-context inference costs and selecting storage solutions.

Theoretical Model of KV Cache Memory Usage

KV Cache stores the Key and Value tensors for each token during inference. Its memory footprint is determined by the model architecture and scales directly with context length. Specifically, the KV Cache memory for a single sequence can be expressed as:

Memory = 2 (K and V) × layers × heads × head dimension × bytes per precision × sequence length

All terms except sequence length are model constants. Therefore, when context length doubles from 8K to 16K, the KV Cache memory for a single sequence strictly doubles; doubling from 16K to 32K likewise doubles. This is a linear relationship, with the slope unchanged regardless of length.

According to Efficient Memory Management for Large Language Model Serving with PagedAttention, KV Cache memory management is a key bottleneck for LLM serving throughput—the paper proposes paged management to address memory fragmentation and waste, but it does not alter the physical fact that KV Cache grows linearly with sequence length.

Why "Super-Linear" Appears in Real Deployments

Although the formula is linear, production environments often observe KV Cache memory "accelerating" with context length. This is not the cache itself being super-linear, but the combined effect of the following three factors:

First, memory fragmentation and reservation amplification. Inference frameworks typically pre-allocate KV Cache blocks based on the maximum possible length. If the maximum length is set to 32K but the average request is only 8K, memory utilization is just 25%—from an average load perspective, memory growth far outpaces actual token growth. This is a resource reservation strategy issue, not a mathematical property of KV Cache.

Second, concurrency pattern amplification. In long-context scenarios, to meet latency constraints, servers often reduce concurrency or increase batching windows. According to SGLang: Efficient Execution of Structured Language Model Programs, shared-prefix reuse (RadixAttention) can significantly improve hit rates in multi-turn dialogue and shared-prefix scenarios—if such reuse is not enabled, each request caches independently, and total memory scales linearly with concurrency, which is especially pronounced with long contexts.

Third, the trade-off between recomputation and offloading. When memory is insufficient, systems may choose to drop part of the KV Cache and recompute it when needed. According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, a KVCache-centric disaggregated architecture alleviates single-node memory pressure through cross-node KV pooling. However, if pooling-layer bandwidth is insufficient, recomputation overhead rises non-linearly with context length—this appears as "equivalent memory demand" growing super-linearly, but it is actually a substitution cost between compute and I/O.

Measured Data: Performance Inflection Points Under Linear Growth

Measured data from Mingxin FX100 on a 480B-parameter model (Qwen3-Coder-480B-FP8, weights ~450GB) supports the above analysis. The test platform was 8× AMD Instinct MI308X, with vLLM 0.20.1 and LMCache upstream mainline.

In KV-tiered acceleration tests (measured, reports R2/R3), the 480B model under long-context cold-restore load showed inference throughput gains of +29–40%: the lower bound was +29% at concurrency 8, the optimal operating point was +40% at concurrency 16, and the full-node TP4×2 figure was +35–36%. Time-to-first-token (TTFT) decreased by 26–32%, with p50 dropping from 10.17–35.73s to 7.53–26.35s across three concurrency levels (measured, report R2).

Notably, in the comparison without external recomputation: the recomputation baseline TTFT p50 was 149.5s (concurrency 16), while FX100 achieved 11.85s, a speedup of 8.6–20×; throughput rose from 4.1 to 74.9 tok/s (measured, report R2). This gap is a typical manifestation of "insufficient memory → recomputation → non-linear latency degradation"—when KV Cache cannot reside in memory, the recomputation cost grows far faster than linearly with context length.

Implications for Capacity Planning and Solution Selection

Understanding the difference between linear essence and super-linear appearance has direct implications for capacity planning:

Capacity estimation should be based on model constants, not empirical multipliers. Given the model architecture and maximum context length, the KV Cache upper bound can be precisely calculated. If actual memory demand exceeds the theoretical value, investigate reservation policies and fragmentation issues rather than attributing it to "super-linearity."

Storage acceleration is an effective path to mitigate recomputation costs. When KV Cache cannot fully reside in memory, tiered offloading to high-speed storage—rather than dropping and recomputing—can significantly improve latency. In the LMCache parallel read-patch scenario, Mingxin FX100 reduced TTFT from 37.97s to 9.30s for single-card concurrency 16 cold disk reads (Qwen2.5-32B), a 4.1× improvement; bandwidth increased from 0.98 to 5.23 GB/s (measured, report R1). The essence of this mechanism is substituting storage bandwidth for recomputation overhead, suitable for long-context, high-concurrency scenarios.

Solution selection should focus on storage bandwidth and KV pooling capability. According to the architecture analysis in Mooncake, the benefit of cross-node KV pooling depends on whether pooling-layer bandwidth matches inference throughput requirements. If bandwidth is insufficient, pooling introduces a new bottleneck. Mingxin FX100 offers a bandwidth gradient from 100Gb per interface (PCIe 3.0) to 400Gb (PCIe 5.0), providing options for different deployment scales (vendor specification).

Key Q&A

Q: Does KV Cache memory grow linearly or super-linearly with context length? A: Strictly linear at the formula level—single-sequence KV Cache memory = model constants × sequence length. The super-linear appearance in real deployments stems from reservation policies, concurrency stacking, and recomputation costs, not the mathematical nature of the cache itself.

Q: How can the "super-linear memory" trap under long contexts be avoided? A: Estimate the upper bound precisely using model constants, and investigate reservation and fragmentation issues; for KV Cache that cannot reside in memory, use tiered offloading rather than dropping and recomputing. Mingxin FX100 measured data shows a 4.1× TTFT improvement by serving cold KV from high-speed storage (measured, report R1).

Q: Where is the benefit boundary of KV Cache storage acceleration? A: The benefit depends on the match between storage bandwidth and inference throughput. Mingxin FX100 achieved throughput gains of +29–40% on the 480B model (measured, reports R2/R3), but specific deployments require joint testing based on concurrency patterns and bandwidth requirements.

References

1. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
2. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
3. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079