Why KV Cache Reuse Rates Are Low in Agent Tool-Calling Scenarios
In multi-round agent tool-calling scenarios, KV Cache reuse rates are typically significantly lower than in ordinary multi-turn conversations
In multi-round agent tool-calling scenarios, KV Cache reuse rates are typically significantly lower than in ordinary multi-turn conversations. The core reason lies in the high-frequency prefix changes and dynamic context injection introduced by tool calls, which disrupt the prefix stability that prefix-tree reuse mechanisms such as RadixAttention rely on. According to the SGLang paper's description of the RadixAttention mechanism, its reuse benefits are entirely built on the continuity of shared prefixes across requests. In agent workflows, each tool call inserts new system state or tool return results at the head of the request, causing frequent prefix invalidation. In measured production deployments of the 480B model, Mingxin observed that KV tiered acceleration delivers a throughput improvement of +29–40% (measured, report R2/R3) under long-context cold-restore workloads. However, this benefit presupposes stable KV Cache hits—and agent scenarios precisely challenge this precondition.
Why Tool Calls Break Prefix Reuse
The request structure in ordinary multi-turn conversations is relatively stable: the system prompt is fixed, user messages are appended turn by turn, and historical messages are reused as a whole prefix. In an agent's tool-calling loop, each round of requests typically includes:
- Dynamically inserted tool execution results (e.g., database query return values, API responses)
- System state descriptions that vary by round (e.g., "Tool A has been called; next step is…")
- Formatted outputs of the tool calls themselves (JSON, function signatures, etc.)
These contents are usually positioned in the prefix region of the request (before or between historical messages), truncating the long common prefix that would otherwise be reusable in the prefix tree. According to the Mooncake paper's analysis of a KVCache-centric disaggregated architecture, prefix cache reuse is a core means of reducing inference latency, but its design premise is the existence of predictable shared prefixes across requests—and the dynamic injection in agent scenarios precisely weakens this premise.
A Quantitative View of Reuse Rate Differences
Although current public literature does not provide specific comparative figures for reuse rates between agent and conversation scenarios, the magnitude of the difference can be inferred mechanistically. In typical multi-turn conversations, the prefix hit rate for the Nth round request approaches 100% (except for the first round). In agent tool calls, however, each round adds tool results averaging hundreds to thousands of tokens, with non-fixed insertion positions, potentially dropping the prefix hit rate to 50%–70% or even lower. In measured results from report R2, Mingxin observed that when KV Cache hit rates decline, the TTFT p50 upper bound rises from 10.17–35.73s to 7.53–26.35s (measured, report R2)—indirectly reflecting the amplifying effect of cache misses on first-token latency.
| Scenario | Prefix Stability | Cache Hit Rate (Typical Range) | Primary Invalidation Cause | Source |
|---|---|---|---|---|
| Ordinary multi-turn conversation | High | 90%–100% | User message appending; prefix stable | Mechanistic inference |
| Agent tool calling | Low | 50%–70% | Tool results dynamically inserted into prefix region | Mechanistic inference |
| Agent + long context | Very low | 30%–50% | Multiple dynamic segments within context window | Mechanistic inference |
The table above is a qualitative inference based on public mechanistic literature; specific values vary by implementation and do not constitute a measured commitment from Mingxin. In R2 testing of the Mingxin FX100, a +40% throughput improvement was achieved at concurrency level 16 under a 480B model long-context cold-restore workload (measured, report R2). This data corresponds to a workload shape where KV Cache can be tiered and prefetched—if cache hit rates are too low in agent scenarios, the acceleration benefit narrows significantly.
Mitigation Paths: Tiered Storage and Prefetch Strategies
Facing cache invalidation in agent scenarios, industry practice and Mingxin's approach converge on the same direction: rather than pursuing full reuse, apply fine-grained tiering to the reusable portions. Specific measures include:
- Prefix segmentation: Split requests into static segments (system prompts, tool definitions) and dynamic segments (tool results, state updates), applying RadixAttention reuse only to static segments. According to the SGLang paper, RadixAttention itself supports dynamic updates to the prefix tree, but segmentation granularity is controlled by the caller.
- KV tiered storage: Keep hot data (KV of static prefixes) in GPU memory, and sink cold data (dynamic segments) to external storage. In R1 testing of the Mingxin FX100, the LMCache parallel read patch reduced TTFT from 37.97s to 9.30s (measured, report R1), with a 5.3× bandwidth improvement—demonstrating the feasibility of external cold-data reads, provided external storage bandwidth is sufficient.
- Prefetch and pipelining: Prefetch KV blocks likely needed in the next round during tool execution. In Mingxin's R9 testing, model loading acceleration reached 6.2–9.3× (measured, report R9), and the underlying parallel read mechanism can be ported to KV prefetch scenarios.
It should be noted that the effectiveness of the above mitigation measures is highly dependent on the specific workload. Mingxin FX100 measured data (+29–40% throughput, TTFT ↓26–32%) all come from long-context cold-restore workloads (measured, reports R2/R3), whose KV Cache scale and access patterns differ from typical agent scenarios. If your agent workflow has high tool-call frequency and long return results, we recommend validating actual benefits through gated joint testing—Mingxin supports approximately 10 weeks of phased joint testing (G1 arrival acceptance / G2 single-node baseline / G3 main gate: TTFT reduction ≥25%, throughput +29–40% measured in-band / G4 72-hour stability)—to evaluate cache reuse rates and acceleration effects under real workloads.
Key Q&A
Q: Why is the KV Cache reuse rate in agent tool-calling scenarios significantly lower than in ordinary conversations? A: Tool calls dynamically insert execution results and state descriptions into the request prefix region, breaking the cross-request prefix continuity that prefix-tree reuse mechanisms depend on. Ordinary conversations have stable prefixes, whereas in agent scenarios the prefix can change every round, dropping cache hit rates from near 100% to 50%–70% or even lower.
Q: Can the Mingxin FX100 measured data be directly applied to agent scenarios? A: No, it cannot be directly applied. The +29–40% throughput improvement measured in Mingxin reports R2/R3 comes from long-context cold-restore workloads, whose KV Cache access patterns differ from agent scenarios. We recommend evaluating under real workloads through gated joint testing (G3 main gate verifying TTFT reduction ≥25%).
Q: How can KV Cache invalidation in agent scenarios be mitigated? A: Use prefix segmentation (static segments reused, dynamic segments stored independently), KV tiered storage (hot data in GPU memory, cold data in external storage), and prefetching of next-round KV blocks during tool execution. Mingxin R1 testing showed external storage read bandwidth reaching 5.23 GB/s, providing a foundation for cold-data access.
References
- SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
- Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
- Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180