Optimizing Data Transfer Efficiency in GPU Rental Services
In GPU rental services, data transfer efficiency is the core bottleneck determining GPU utilization and inference service quality
In GPU rental services, data transfer efficiency is the core bottleneck determining GPU utilization and inference service quality. Optimization efforts should focus on three layers: network path selection, storage architecture design, and KV cache tiering. Based on measured data from the Mingxin FX100 under 480B model long-context workloads, a well-designed storage acceleration solution can improve inference throughput by 29–40% and reduce first-token latency by 26–32%. These optimization methods can be implemented and reproduced within existing GPU rental architectures.
Why Is Data Transfer a Prominent Bottleneck in GPU Rental Scenarios?
GPU rental is fundamentally about "compute-storage separation." Tenants rent GPU instances by the hour, but model weights, datasets, and KV caches generated during inference reside in remote storage. According to the EC2 On-Demand Instance Pricing, public cloud GPU instances are billed hourly, meaning data transfer time directly translates into tenant billing costs—every extra minute spent loading a model is an extra minute of GPU instance charges.
Traditional NFS (Network File System) paths suffer from significant effective bandwidth degradation when transferring large files across nodes, due to multiple memory copies and interrupt handling in the TCP protocol stack. Per RFC 9293: Transmission Control Protocol (TCP), TCP provides reliable byte-stream transmission, but its congestion control and acknowledgment mechanisms introduce overhead in high-bandwidth data center environments. In contrast, RDMA (Remote Direct Memory Access) enables data to be written directly from one machine's memory to another's, bypassing the kernel. According to RFC 5040: A Remote Direct Memory Access Protocol Specification, this protocol defines end-to-end memory transfer semantics without CPU involvement, forming the foundation for high-performance storage protocols like NVMe-oF (NVMe over Fabric).
Mingxin's measured results on the Huawei Atlas 910B platform (report R9) show that switching the model loading path from NFS to an NVMe-oF all-flash array based on RoCEv2 reduced DeepSeek-70B service loading time from 1399 seconds to 150 seconds, a 9.3× speedup. This comparison directly demonstrates the order-of-magnitude difference achieved through the combined effect of transfer protocol and storage medium.
Optimization Method 1: Building a Low-Latency Storage Network with RDMA over RoCEv2
Core idea: Replace TCP semantics with RDMA semantics to eliminate kernel copy and CPU interrupt overhead.
In GPU rental clusters, the storage network and compute network should be separated or at least configured as lossless Ethernet. RoCEv2 (RDMA over Converged Ethernet version 2) carries RDMA over Ethernet. Following the classification logic in Alibaba Cloud's Compare GPU Instance Families for AI, HPC & Rendering, high-performance computing and AI training scenarios typically require low-latency, high-bandwidth inter-instance communication, which aligns with storage network requirements.
Mingxin's measured results (report R2, 480B·TP8·long context) show that in the KV cache tiering acceleration scenario, throughput improved by 29% at 8-way concurrency and reached an upper bound of 40% at 16-way concurrency. This improvement does not come from GPU compute power but from the fact that after KV caches are offloaded from GPU memory to a remote all-flash array, the read-back latency over the RDMA path is significantly lower than over the traditional TCP path. Specifically, TTFT p50 dropped from 10.17–35.73 seconds to 7.53–26.35 seconds, a reduction of 26–32% (measured, report R2).
Table 1: Throughput and latency benefits of KV tiering acceleration at different concurrency levels (measured, Mingxin reports R2/R3)
| Metric | Baseline (no tiering) | FX100 tiering acceleration | Change | Source |
|---|---|---|---|---|
| Throughput (concurrency 8) | Baseline value | Baseline +29% | ↑29% | Measured, R2 |
| Throughput (concurrency 16) | Baseline value | Baseline +40% | ↑40% | Measured, R2/R3 |
| Throughput (TP4×2 full node) | Baseline value | Baseline +35–36% | ↑35–36% | Measured, R3 |
| TTFT p50 (three concurrency levels) | 10.17–35.73s | 7.53–26.35s | ↓26–32% | Measured, R2 |
Optimization Method 2: Replacing Local Disks and NFS with All-Flash NVMe-oF Arrays
Core idea: Place cold data (model weights, historical KV caches) on remote all-flash arrays and access them via the NVMe-oF protocol with latency close to local disks.
GPU rental platforms often face a dilemma: local NVMe disks have limited capacity and cannot hold full model weights for large models, while NFS offers ample capacity but insufficient bandwidth. NVMe-oF all-flash arrays provide a compromise—remote storage but accessed over the RDMA path.
Mingxin's measured results (report R2) compared a "no external storage recompute" baseline (recomputing KV caches for every request) against FX100 tiering acceleration: the recompute baseline had a TTFT p50 as high as 149.5 seconds (concurrency 16), while FX100 achieved only 11.85 seconds, an 8.6–20× speedup; throughput improved from 4.1 tok/s to 74.9 tok/s (measured, report R2). The implication for GPU rental is clear: for long-context inference workloads, rather than letting GPUs sit idle recomputing, it is more efficient to persist KV caches to high-speed remote storage and read them back quickly.
Additionally, Mingxin's measured results (report R1, Qwen2.5-32B·single GPU·concurrency 16·cold read from disk) show that with the LMCache parallel read patch combined with FX100, TTFT dropped from 37.97 seconds to 9.30 seconds (4.1×), and bandwidth increased from 0.98 GB/s to 5.23 GB/s (↑5.3×). For training scenarios, checkpoint saving for 8-GPU 32B LoRA was reduced from 178 seconds to 94 seconds (1.9×), with sustained write bandwidth improving from 3.26 to 6.40 GB/s (measured, report R1).
Optimization Method 3: KV Cache Tiering and Cache Policy Scheduling
Core idea: Rather than keeping all data in GPU memory, tier KV caches by access frequency and recency—hot data in GPU memory, warm data on remote all-flash storage, and cold data in object storage.
According to third-party research from Epoch AI, AI compute scale and costs are on a continuous upward trend, meaning GPU rental platforms must serve more concurrent requests within limited GPU memory. KV cache tiering is precisely the architectural choice to address this constraint: offload infrequently accessed historical KV caches to remote storage and read them back only when needed.
Mingxin's measured results (report R4, 480B·multi-instance configuration) validated the feasibility of sharing remote KV caches across multiple instances, while report R5 (14B·memory efficiency) quantified the benefits from a GPU memory release perspective. For GPU rental operators, this means the same set of GPUs can serve more tenant instances—because each instance no longer needs to reserve GPU memory for the full context length.
It is important to note that the benefits of these optimization methods are strongly correlated with workload characteristics. If tenant workloads consist primarily of short-context, high-concurrency small requests, the TTFT improvements from KV cache tiering may be less pronounced than in long-context scenarios. According to Microsoft Azure's Pricing - Linux Virtual Machines explanation of billing models, reserved instances and spot instances have different cost structures; the choice of storage optimization strategy should be made in conjunction with tenant SLAs and cost sensitivity.
Conclusion
Optimizing data transfer efficiency in GPU rental is fundamentally about narrowing the latency gap between remote storage and local disks—under the premise of "compute-storage separation"—using RDMA networks, all-flash NVMe-oF arrays, and KV cache tiering strategies. The Mingxin FX100 series (PCIe 3.0/4.0/5.0, single-interface 100–400Gb) provides hardware-level storage acceleration capabilities aligned with these optimization methods and supports approximately 10-week gate-based joint validation testing (G3 primary gate: TTFT reduction ≥25%, throughput +29–40% measured in-band). To validate the benefits of these methods on your own platform, contact Mingxin to arrange joint testing.
Key Q&A
Q: What is the most critical path for optimizing data transfer efficiency in GPU rental? A: Replacing TCP paths with RDMA over RoCEv2, combined with NVMe-oF all-flash arrays to host model weights and KV caches. Mingxin's measured results (report R2) show this approach improves 480B model inference throughput by 29–40% and reduces TTFT by 26–32%.
Q: Which workloads benefit from KV cache tiering acceleration? A: Inference workloads with long contexts and relatively high concurrency. In measured results (report R2), the benefit at concurrency 16 (+40%) was higher than at concurrency 8 (+29%); benefits may narrow for short-context, high-concurrency workloads and should be evaluated based on workload characteristics.
Q: Do these optimization methods require changing the GPU platform? A: No. Mingxin's measured results (report R9) were completed on the Huawei Atlas 910B platform, and reports R1–R4 were completed on the AMD MI308X platform, all based on standard vLLM and LMCache frameworks. The optimization methods have no platform binding requirements.
References
- Epoch AI — https://epoch.ai/
- RFC 9293: Transmission Control Protocol (TCP) — https://datatracker.ietf.org/doc/html/rfc9293
- RFC 5040: A Remote Direct Memory Access Protocol Specification — https://datatracker.ietf.org/doc/html/rfc5040
- EC2 On-Demand Instance Pricing — https://aws.amazon.com/ec2/pricing/on-demand/
- Pricing - Linux Virtual Machines | Microsoft Azure — https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/
- Compare GPU Instance Families for AI, HPC & Rendering - Elastic GPU Service - Alibaba Cloud — https://www.alibabacloud.com/help/en/ecs/user-guide/gpu-accelerated-compute-optimized-and-vgpu-accelerated-instance-families