NFS Bottleneck: 691s vs 112s Inference Model Loading
NFS, as the default shared storage option for inference clusters, is becoming the most insidious killer of GPU utilization in model loading scenarios
NFS, as the default shared storage option for inference clusters, is becoming the most insidious killer of GPU utilization in model loading scenarios. Measured data on the Huawei Atlas 910B platform shows that DeepSeek-32B model service loading time dropped from a 691-second NFS baseline to 112 seconds, and DeepSeek-70B dropped from 1399 seconds to 150 seconds, achieving speedups of 6.2x and 9.3x respectively [source: measured, report R9 (Ascend platform)]. This gap means that in rolling updates or failure recovery scenarios for a thousand-GPU cluster, NFS can leave GPU arrays worth millions of dollars idle for minutes to tens of minutes. This article breaks down the bottleneck mechanisms of NFS based on measured data and provides verifiable alternative paths.
Three Structural Causes of the NFS Loading Bottleneck
Protocol stack overhead and metadata lock contention. NFS (Network File System) is designed around RPC semantics; each file read traverses multiple layers including client cache, protocol encapsulation, server-side metadata lookup, and data copy. When multiple GPU nodes load the same model weights simultaneously, the metadata lock (inode lock) on the NFS server becomes a serialization bottleneck. In measurements, DeepSeek-70B weights are approximately 140GB, and loading under NFS took 1399 seconds, with average throughput below 100 MB/s—far below the bandwidth ceiling of a 10GbE network.
Small-file random read amplification. Model weight files are typically stored in sharded formats (e.g., safetensors), and within each shard, tensor alignment creates gaps. NFS's 4KB or 64KB block mapping mechanism generates significant useless prefetching in random read scenarios, further reducing effective throughput. In contrast, the NVMe-oF (NVMe over Fabric) protocol directly maps the namespace of remote NVMe devices, making block addressing identical to local NVMe and eliminating read amplification at the file system layer.
TCP/IP protocol stack and lack of GPU Direct. Traditional NFS runs over the TCP/IP protocol stack, requiring data to pass through kernel buffers, page cache, and user-space copies at multiple levels, and cannot leverage RDMA or GPUDirect RDMA capabilities. On the Ascend 910B platform, the NFS client must first read data into CPU memory, then copy it to NPU memory via the HCCS bus, adding two memory copies along the path. In contrast, the Mingxin FX100 all-flash NVMe-oF array supports RoCEv2 and RDMA, allowing data to be written directly from the storage side to GPU memory via RDMA, bypassing the CPU memory intermediary.
Measured Comparison: Relationship Between Loading Time and Cluster Utilization
On the Huawei Atlas 910B platform (R9 test report), the test environment was an 8-card 910B node, comparing model loading performance between NFS and the Mingxin FX100:
| Model | NFS Loading Time | FX100 Loading Time | Speedup |
|---|---|---|---|
| DeepSeek-32B | 691 s | 112 s | 6.2x |
| DeepSeek-70B | 1399 s | 150 s | 9.3x |
Taking the 70B model as an example, loading under NFS takes 23.3 minutes. Assuming a cluster of 64 inference nodes with 8 cards per node, a single rolling update (each node restarting and loading sequentially) results in total idle GPU time of 64 × 23.3 minutes ≈ 24.8 hours. With the FX100, per-node loading drops to 2.5 minutes, reducing total idle time to approximately 2.7 hours—an approximately 9x improvement in GPU utilization. This calculation does not yet account for additional loading cycles from failure recovery (e.g., automatic restarts after OOM).
In KV Cache offloading scenarios, the impact of the loading bottleneck is even more pronounced. Measured data from R2 shows that under a 480B model long-context cold-restore workload, the FX100 reduced time-to-first-token (TTFT) from 10.17–35.73 seconds to 7.53–26.35 seconds (a 26–32% reduction) [source: measured, report R2]. If NFS were used as remote storage for the KV Cache, TTFT would degrade further due to network latency and protocol overhead, potentially causing inference service timeouts.
Two Paths to Efficiency Optimization: Storage Protocol Upgrade and Tiered Caching
Path one: NVMe-oF as a replacement for NFS as the shared storage protocol. The Mingxin FX100 uses NVMe-oF over RoCEv2, presenting remote NVMe devices as block devices to clients, with support for RDMA and GPUDirect RDMA. In measurements, the FX100 achieved 6–9x the loading throughput of NFS on the Ascend platform, with lower latency jitter. For clusters with existing NFS infrastructure, NFS can be retained for management-plane storage while model weights and KV Cache are migrated to a high-performance NVMe-oF storage pool.
Path two: tiered KV Cache caching strategy. Even with a storage protocol upgrade, cold-start loading remains unavoidable. Measured data from R1 shows that the LMCache parallel-read patch combined with the FX100 reduced per-card cold-read TTFT from 37.97 seconds to 9.30 seconds (a 4.1x improvement), with bandwidth increasing from 0.98 GB/s to 5.23 GB/s [source: measured, report R1]. This demonstrates that combining low-latency read capability on the storage side with parallel scheduling at the caching software layer can further compress the cold-start window.
For training scenarios, checkpoint saving is similarly constrained by NFS write performance. In R1 measurements, saving a full-model snapshot for an 8-card 32B LoRA (65.6GB per copy) dropped from 178 seconds to 94 seconds, a 96% improvement in sustained write bandwidth [source: measured, report R1]. This directly shortens recovery time after training interruptions, reducing GPU wait time.
Conclusion
The NFS bottleneck is fundamentally a matter of protocol stack overhead and lack of hardware offload capability, not insufficient network bandwidth. Measured data shows that adopting the NVMe-oF protocol with RDMA offload can reduce model loading time by 6–9x, translating directly into improved GPU utilization. Mingxin Technology offers the FX series all-flash storage arrays based on NVMe-oF, supporting joint testing with mainstream compute platforms including Ascend and AMD. We welcome compute centers and model service providers to conduct gate-based joint testing (approximately 10 weeks, including a primary gate metric of TTFT reduction ≥25%), validating efficiency gains with reproducible measured data.
Key Q&A
Q: Why is NFS model loading 6–9x slower than NVMe-oF?
A: NFS is based on RPC and the TCP/IP protocol stack, suffering from metadata lock contention, small-file read amplification, and CPU memory intermediary overhead; NVMe-oF uses RDMA to directly map remote NVMe block devices, bypassing the file system layer and CPU copies, achieving 6.2–9.3x loading acceleration measured on the Huawei 910B platform [source: measured, report R9].
Q: How significant is the impact of model loading time on GPU utilization?
A: Taking a 64-node cluster rolling update as an example, the 70B model takes 23.3 minutes per node under NFS, totaling approximately 24.8 hours of idle time; with the FX100, this drops to 2.5 minutes per node, totaling approximately 2.7 hours of idle time—an approximately 9x improvement in GPU utilization.
Q: Besides protocol upgrades, what other efficiency optimization methods exist?
A: A tiered KV Cache caching strategy (e.g., the LMCache parallel-read patch) can shorten cold-read TTFT by 4.1x [source: measured, report R1]; training checkpoint saving can be accelerated by 1.9x [source: measured, report R1]; both work synergistically with NVMe-oF storage.