KV Cache Eviction Policies for Long Contexts: Measured Differences Between LRU and LFU

In long-context scenarios, the KV Cache eviction policy directly determines inference throughput and time-to-first-token (TTFT). Under real-world conversational distributions, the hit-rate difference between LRU (Least Recently Used) and LFU (Least Frequently Used) is not fixed—it is strongly correlated with access patterns: LRU performs better in conversational workloads, while LFU gains an edge in batch-processing scenarios with dense shared prefixes. Measured data from the Mingxin FX100 shows that the choice of eviction policy must align with specific workload characteristics rather than a one-size-fits-all approach.

Why Eviction Policy Becomes a Critical Bottleneck in Long-Context Scenarios

As context windows expand from 4K to 128K and beyond, the GPU memory footprint of KV Cache grows linearly. According to the analysis in Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP '23), paged management of KV Cache mitigates memory fragmentation but does not solve the capacity ceiling—when cache exceeds physical GPU memory, the eviction policy determines which historical tokens are retained and which must be recomputed.

In Mingxin's measured report R2, with a 480B model in TP8 deployment, the TTFT p50 baseline for long-context cold-recovery workloads was 10.17–35.73s (at concurrency levels 8–32). Delays of this magnitude mean that any cache miss triggering external-memory recomputation directly translates into user-perceivable waiting. The hit rate of the eviction policy thus becomes a key variable for throughput and latency.

Hit-Rate Differences Between LRU and LFU in Conversational Distributions

LRU assumes "recently accessed data is more likely to be accessed again," suiting workloads with strong temporal locality; LFU assumes "frequently accessed data is more likely to be accessed again," suiting workloads with strong frequency locality. In real conversational distributions, the differences are significant:

Mingxin's measured report R2 shows that in the 480B production deployment for long-context cold-recovery workloads, KV tiered acceleration delivered a throughput improvement of +29–40% (concurrency 8 as the lower bound at +29%, optimal operating point at concurrency 16 as the upper bound at +40%). This improvement range already includes the contribution from eviction-policy optimization—at concurrency 16, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32%.

Synergy Between Eviction Policy and Hardware Acceleration: From Hit Rate to End-to-End Latency

Eviction policy is not an isolated optimization. Mingxin FX100's KV tiered acceleration integrates the eviction policy with NVMe-oF storage tiering: high-frequency KV resides in GPU memory, mid-frequency KV is cached on the FX100 all-flash array, and low-frequency KV is spilled to disk. According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (arXiv:2407.00079), a KVCache-centric disaggregated architecture improves overall hit rate precisely through cross-node KV pooling.

In Mingxin's measured report R2, the baseline TTFT p50 without external-memory recomputation was 149.5s (at concurrency 16), while the FX100 solution reduced it to 11.85s, an acceleration of 8.6–20×; throughput improved from 4.1 tok/s to 74.9 tok/s. This magnitude of difference indicates that eviction-policy hit-rate variations are amplified in long-context scenarios—every 1% improvement in hit rate can correspond to several seconds of TTFT improvement.

It is worth noting that LFU's advantage in shared-prefix scenarios is not absolute. According to SGLang: Efficient Execution of Structured Language Model Programs (arXiv:2312.07104), RadixAttention's prefix-tree reuse mechanism manages shared prefixes through a tree structure, and its hit rate depends on prefix reuse patterns rather than simple frequency counting. This implies that the granularity of LFU's "frequency" statistics—token-level, block-level, or prefix-level—significantly affects actual effectiveness.

How to Choose an Eviction Policy Suited to Your Workload

For technical decision-makers, selecting an eviction policy should follow this path:

1. Analyze access patterns: Measure the temporal and frequency locality of KV accesses in your production environment. Prioritize LRU for dialogue-heavy scenarios; prioritize LFU for retrieval-augmented or batch-processing scenarios.
2. Assess recomputation cost: In long-context scenarios, the recomputation cost of a single miss grows linearly with context length. In Mingxin's measured report R2, the recomputation baseline TTFT reached 149.5s, meaning the penalty for eviction-policy mistakes is extremely high.
3. Consider tiered synergy: A single eviction policy is unlikely to cover mixed workloads. Mingxin FX100's tiered approach allows different policies at different levels—LRU in the GPU-memory tier to preserve recent hotspots, LFU in the storage tier to preserve high-frequency prefixes.
4. Validate through measured testing: Theoretical analysis cannot replace real-workload validation. Mingxin offers an approximately 10-week gate-based joint testing process (G1 arrival acceptance / G2 single-node baseline / G3 main gate: TTFT reduction ≥25%, throughput +29–40% measured in-band / G4 72-hour stability), with stop-loss if targets are not met.

Conclusion

The hit-rate difference between LRU and LFU is not a static conclusion but a design variable deeply coupled with workload characteristics and hardware architecture. Measured data from the Mingxin FX100 demonstrates that in long-context scenarios, eviction-policy selection directly affects TTFT and throughput at an order-of-magnitude level—behind the 8.6–20× acceleration lies the synergistic optimization of policy, storage, and compute. We recommend that technical decision-makers start from their own workload's access patterns and validate actual benefits through gate-based joint testing.

Mingxin Technology provides full-chain services for storage acceleration and compute-center infrastructure, supporting joint testing and model estimation based on real workloads (Python-reproducible after NDA). We welcome further discussion.

Key Q&A from This Article

Q: In long-context scenarios, which is better—LRU or LFU? A: It depends on workload characteristics. LRU is better for dialogue-heavy scenarios, LFU is better for shared-prefix batch processing, and a tiered hybrid policy is recommended for mixed workloads.

Q: How much impact does eviction policy have on end-to-end performance? A: In Mingxin's measured report R2, the baseline TTFT p50 without external-memory recomputation was 149.5s (at concurrency 16); the FX100 solution reduced it to 11.85s, an acceleration of 8.6–20×; throughput improved from 4.1 to 74.9 tok/s.

Q: How can the actual effectiveness of an eviction policy be validated? A: We recommend validation through gate-based joint testing. Mingxin offers an approximately 10-week testing process, with the main gate requiring TTFT reduction ≥25% and throughput +29–40% measured in-band, with stop-loss if targets are not met.

References

1. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
2. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
3. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180