External KV Tiering vs Local NVMe: Gap Analysis

Key takeaway: Under long-context cold-restore workloads on a 480B-class MoE model, external KV tiering (Mingxin FX100) reduces time-to-first-token (TTFT) by 26–32% relative to a local NVMe baseline, and delivers 8.6–20× speedup over a no-external-storage recompute path. For model loading, external storage achieves 6.2–9.3× improvement over an NFS baseline. The gaps among the three paths are not linearly additive; they depend on where the bottleneck lies—when the bottleneck is KV read bandwidth, external tiering shows a clear advantage; when the bottleneck is compute or network, the gap narrows.

Why Compare These Three Paths

The core tension in long-context inference (e.g., 480B parameters, TP8 deployment) is that KV Cache grows linearly with sequence length, while per-GPU HBM capacity is limited. When KV cannot fully reside in GPU memory, the system must choose among the following paths:

1. No external storage, recompute: KV is not saved; each request recomputes prefill from scratch. The cost is wasted compute, and TTFT degrades linearly with concurrency.
2. Local NVMe offload: KV is written to node-local NVMe and read via PCIe. Latency is controllable, but bandwidth is limited by single-drive and PCIe channel constraints.
3. External KV tiering (e.g., Mingxin FX100): KV is tiered on an NVMe-oF array, accessed over RoCEv2 networking, with KV indexing and prefetch handled on the storage side.

The essence of the gap among the three is the trade-off between "compute recompute cost" vs "storage read bandwidth." All data below comes from the formal test report of Mingxin FX100 on an 8×AMD MI308X platform (ROCm 7.2, vLLM 0.20.1+rocm721, LMCache upstream mainline).

Gap 1: TTFT and Throughput — External Tiering vs Local NVMe vs Recompute

Under 480B·TP8 long-context cold-restore workloads at three concurrency levels (8/16/32), measured data from report R2 shows:

• External tiering vs local NVMe: TTFT p50 drops from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32% (measured, report R2). Throughput improves by 29–40% (measured, reports R2/R3; the lower bound of 29% is at concurrency 8, the upper bound of 40% is at the optimal operating point at concurrency 16).
• External tiering vs no-external-storage recompute: The gap is far more pronounced. The recompute baseline TTFT p50 is 149.5s (at concurrency 16), while FX100 achieves 11.85s—a 12.6× speedup. Throughput rises from 4.1 tok/s to 74.9 tok/s, an 18.3× improvement. Across different operating points, the overall speedup range is 8.6–20× (measured, report R2).

Interpretation: The gap between local NVMe and no-external-storage recompute is essentially the gap between "reading from disk" and "recomputing." At 480B scale, the compute cost of recomputing prefill for a 32K context far exceeds the I/O cost of reading KV from storage. The advantage of external tiering over local NVMe comes from storage-side parallel prefetch and index optimization for KV—local NVMe is limited by single-drive sequential read bandwidth and requires host CPU cycles to handle the I/O stack.

Gap 2: Model Loading and Checkpoint — The "Hidden Benefit" of Storage Acceleration

Beyond the inference path, storage performance also affects service loading and training interruption recovery. Measured data from report R9 (Huawei Atlas 910B platform) shows:

• Model loading (external vs NFS): DeepSeek-32B service loading drops from 691s to 112s (6.2×); DeepSeek-70B drops from 1399s to 150s (9.3×). This means that during failure recovery or rolling updates, the service unavailability window shrinks from "minutes" to "seconds."
• Training checkpoint saving (external vs local): For an 8-GPU 32B LoRA run with a 65.6GB full-model snapshot per copy, save time drops from 178s to 94s (1.9×), and sustained write bandwidth rises from 3.26 GB/s to 6.40 GB/s (+96%, measured, report R1).

Interpretation: These benefits are easily overshadowed by KV inference metrics, but they are equally important for production systems. A 1.9× faster checkpoint save shortens the "safety window" during training interruptions, while a 6–9× faster model load directly reduces service availability SLA risk.

Gap 3: LMCache Patch and Single-GPU Scenarios — Amplification Effects at Small Scale

In a single-GPU, concurrency-16, cold-read scenario (Qwen2.5-32B), measured data from report R1 shows: with the LMCache parallel-read patch combined with FX100, TTFT drops from 37.97s to 9.30s (4.1×), and bandwidth rises from 0.98 GB/s to 5.23 GB/s (5.3×).

Interpretation: Even with a 32B small model on a single GPU, the gap between external KV tiering and local disk remains on the order of 4×. This suggests that the gap primarily comes from the storage side's ability to parallelize KV reads, rather than from model scale. The amplification effect in small-model scenarios actually makes them a better starting point for PoC validation.

Selection Guidance: When to Choose Which Path

Based on the measured gaps above, here are three actionable decision paths:

1. If long-context share is <10% and concurrency ≤8: Local NVMe is sufficient; the TTFT gap is within 30% (lower bound, measured, report R2). No need to introduce external storage.
2. If long-context share is >30% or concurrency ≥16: The no-external-storage recompute path is unacceptable (TTFT reaches 149.5s); the gap between local NVMe and FX100 is 26–32% in TTFT and up to 40% in throughput. External tiering becomes a necessary option.
3. If service availability SLA requires recovery time <5 minutes: The 6–9× gap in model loading (measured, report R9) directly determines whether the SLA is met; external storage should be prioritized.

Mingxin FX100 supports approximately 10 weeks of gated joint testing (G1 arrival acceptance / G2 single-node baseline / G3 main gate: TTFT reduction ≥25%, throughput +29–40% measured in-band / G4 72-hour stability), with stop-loss if targets are not met. For technical decision-makers at compute centers, we recommend applying for joint testing with your own models and workload patterns, and using a Python-reproducible estimation model to verify the specific values of the above gaps in your own scenarios.

Q&A: Key Points from This Article

Q: How large is the TTFT gap between external KV tiering and local NVMe? A: Under 480B·TP8 long-context cold-restore workloads, TTFT p50 is reduced by 26–32% (measured, report R2). Throughput improves by 29–40%, with the lower bound at concurrency 8 and the upper bound at the optimal operating point at concurrency 16 (measured, reports R2/R3).

Q: What is the cost of the no-external-storage recompute path? A: Relative to external KV tiering, the recompute path is 8.6–20× slower in TTFT (measured, report R2; recompute baseline 149.5s vs FX100 11.85s). This gap widens as concurrency increases, because the compute consumed by recompute scales linearly with concurrency.

Q: Does external storage benefit model loading and checkpointing? A: Yes. Relative to an NFS baseline, model loading is accelerated by 6.2–9.3× (measured, report R9, Ascend 910B platform); training checkpoint saving is accelerated by 1.9×, with write bandwidth improved by 96% (measured, report R1).