What Does 90% Line-Rate Utilization on a Single 100GbE Port Really Mean: A Network Bottleneck Analysis for Inference Storage

In KV Cache tiered acceleration and model weight loading scenarios, the theoretical bandwidth ceiling for a single 100GbE port is 12.5 GB/s. The Mingxin FX100 achieves a measured bandwidth of 5.23 GB/s (measured, report R1) under LMCache parallel read patching, corresponding to a line-rate utilization of approximately 42%. In training checkpoint saving scenarios, sustained write bandwidth reaches 6.40 GB/s (measured, report R1), with a utilization of about 51%. These figures indicate that line-rate utilization on a single 100GbE port is not the bottleneck metric; the real constraints lie in storage medium latency and protocol stack overhead. This article breaks down the causes of network bottlenecks in inference storage based on measured data from the Mingxin FX100 on AMD MI308X platforms, and provides an evaluation framework for compute center architecture selection.

1. The True Meaning of 100GbE Line-Rate Utilization

A single 100GbE port has a theoretical bandwidth of 12.5 GB/s, but in practice, the NVMe-oF (NVMe over Fabrics) protocol stack itself introduces approximately 5-8% overhead. Combined with RoCEv2's PFC/ECN flow control mechanisms, line-rate utilization typically struggles to exceed 90%. The Mingxin FX100, in a KV Cache cold-read scenario (Qwen2.5-32B, 16 concurrent requests on a single GPU), achieves a measured bandwidth of 5.23 GB/s (measured, report R1), corresponding to a utilization of about 42%. In a training checkpoint saving scenario (8 GPUs, 32B LoRA, 65.6GB snapshots each), sustained write bandwidth reaches 6.40 GB/s (measured, report R1), with a utilization of about 51%.

These two figures fall below the theoretical peak, but this does not imply a network bottleneck. Taking KV Cache cold-read as an example, TTFT drops from 37.97s to 9.30s (measured, report R1), a 4.1x improvement, with bandwidth increasing 5.3x. If the network were the bottleneck, the improvement would be far less significant. The actual bottlenecks are:

• Storage medium latency: Local NVMe single drives (PCIe Gen4) have read latencies of approximately 80-120μs, while NVMe-oF arrays using multi-drive RAID0 can reduce latency to below 50μs.
• Protocol stack overhead: The TCP/IP stack can consume 30-40% of CPU under 100GbE, whereas offloading RoCEv2 to the NIC can reduce CPU usage to below 5%.
• Application-layer scheduling: vLLM's KV Cache management policies and LMCache's cache hit rates directly influence the actual volume of data read.

2. Why 90% Line-Rate Utilization Is Not a Design Target

In inference storage scenarios, pursuing 90% line-rate utilization may actually harm overall performance. There are three reasons:

First, KV Cache reads are latency-sensitive workloads. Under a 480B model long-context cold recovery load, TTFT p50 drops from 10.17-35.73s to 7.53-26.35s (measured, report R2), a reduction of 26-32%. Forcing bandwidth utilization up to 90% would require increasing the concurrent queue depth, which directly adds queuing latency and pushes TTFT higher. The Mingxin FX100 shows a throughput gain of 29% (lower bound) at concurrency level 8 and 40% (upper bound) at concurrency level 16, indicating an optimal operating point exists rather than "the higher, the better."

Second, model loading is bandwidth-sensitive, but the absolute time for a single load matters more. On Huawei Atlas 910B platforms, DeepSeek-70B service loading drops from 1399s to 150s (measured, report R9), a 9.3x speedup. In this case, the 12.5 GB/s theoretical bandwidth of a single 100GbE port is sufficient to transfer 150GB of weights within 2 minutes; the actual bottleneck lies in file system metadata overhead and NFS protocol stack latency, not network bandwidth.

Third, training checkpoint saving is a write-intensive workload that requires balancing bandwidth with stability. The full model snapshot for 8 GPUs with 32B LoRA (65.6GB) drops from 178s to 94s (measured, report R1), with sustained write bandwidth of 6.40 GB/s. Pursuing 90% line-rate utilization here would require enabling larger write caches, which could introduce the risk of data loss on power failure.

3. How to Evaluate Network Bottlenecks in Inference Storage

For compute center architecture selection, we recommend evaluating from three dimensions:

1. Latency-bandwidth joint metrics. Looking at bandwidth utilization alone masks latency issues. We recommend using TTFT reduction and throughput gains as core metrics rather than line-rate utilization. The Mingxin FX100, under 480B·TP8 with three concurrency levels, achieves TTFT p50 reductions of 26-32% (measured, report R2) and throughput gains of 29-40% (measured, reports R2/R3)—metrics that directly reflect user experience.

2. Protocol stack and hardware offload capability. RoCEv2 can reduce CPU usage by over 30% compared to TCP/IP, but requires NIC support. The Mingxin FX100 on AMD MI308X platforms (ROCm 7.2) achieves a measured bandwidth of 5.23 GB/s (measured, report R1), approaching the practical available bandwidth of a single 100GbE port under RoCEv2 (approximately 60-70%).

3. End-to-end scenario validation. We recommend a gate-based joint testing approach with clear pass thresholds. In Mingxin's collaboration model, the G3 primary gate requires TTFT reduction ≥25% and throughput +29-40% (measured in-band), while G4 is a 72-hour stability test with stop-loss criteria if targets are not met. This quantitative validation approach is more reliable than simply examining bandwidth utilization.

Conclusion

A 90% line-rate utilization on a single 100GbE port is neither necessary nor realistic in inference storage scenarios. The true performance bottlenecks lie in storage medium latency, protocol stack overhead, and application-layer scheduling policies. The measured data from the Mingxin FX100 across three core scenarios—KV Cache tiered acceleration, model loading, and checkpoint saving (reports R1/R2/R3/R9)—provides quantifiable reference benchmarks for compute center architecture selection. Teams interested in joint testing are welcome to contact us to validate actual benefits through gate-based testing.

Key Q&A

Q: Does 90% line-rate utilization on a single 100GbE port mean there is no network bottleneck? A: No. The Mingxin FX100 achieves a measured bandwidth of 5.23 GB/s (measured, report R1) in KV Cache cold-read scenarios, with a utilization of about 42%, yet TTFT improves by 4.1x. The bottleneck lies in storage medium latency and protocol stack overhead, not network bandwidth.

Q: In inference storage scenarios, what metrics should be used to evaluate network performance? A: We recommend using TTFT reduction (measured at 26-32% for the Mingxin FX100, report R2) and throughput gains (29-40%, reports R2/R3) as core metrics, rather than line-rate utilization. Also pay attention to RoCEv2 hardware offload capability and end-to-end gate-based validation results.

Q: How does the Mingxin FX100 perform in training checkpoint saving scenarios? A: Full model snapshot saving for 8 GPUs with 32B LoRA drops from 178s to 94s (measured, report R1), with sustained write bandwidth of 6.40 GB/s, a 1.9x speedup. This scenario requires balancing bandwidth with data safety, and pursuing extreme line-rate utilization is not recommended.