Power Composition of Inference Storage and Controllable Optimization Paths
In large language model inference systems, the storage subsystem typically accounts for a far smaller share of power consumption than GPU compute units
In large language model inference systems, the storage subsystem typically accounts for a far smaller share of power consumption than GPU compute units. But this does not mean it deserves no attention—quite the opposite: storage power is one of the few variables in the entire chain that can be actively influenced through architecture selection and technical optimization. GPU power is determined by process node and compute capability, and is largely uncontrollable. In storage power composition, however, data path design, media type selection, and data placement strategies all leave clear room for optimization. Based on measured data from Xinyuanyi FX100 on an AMD MI308X platform (measured, reports R1/R2/R9), this article breaks down the real power composition of inference storage and distinguishes which aspects are truly controllable versus merely theoretical "optimization headroom."
Real Share and Composition of Storage Power in Inference Systems
Power distribution in inference systems is highly imbalanced. GPU accelerators dominate—taking an 8× AMD MI308X test platform as an example, a single card draws hundreds of watts, and the GPU portion alone accounts for the vast majority of total system power (measured, report R2, platform configuration). By contrast, NVMe SSDs typically draw only 5–15W per drive; even with a 4-drive RAID0 array, the storage subsystem's total power is only a few percentage points of system power.
However, a low power share does not mean optimization is worthless. The power problem in inference is fundamentally a "time-integral" problem—energy per inference task = power × duration. As analyzed in the FlashAttention paper, attention computation is limited by HBM bandwidth rather than compute, meaning data movement time often far exceeds compute time in inference (arXiv:2205.14135). Although the storage subsystem has low instantaneous power, if it becomes a bottleneck in data movement, the extended task duration keeps the GPU idling in a high-power state longer, indirectly raising total energy consumption. Measured data from Xinyuanyi FX100 confirms this logic: in a baseline comparison without external-memory recomputation, the recompute baseline TTFT p50 was 149.5s (conc16), while FX100 required only 11.85s (measured, report R2)—storage acceleration sharply reduces GPU wait time for data, lowering the GPU power分摊 per unit of token output.
Which Storage Power Aspects Are Truly Controllable
Data Path: Bypassing the Power and Latency Cost of the CPU
In the traditional storage access path, data read from NVMe drives must first be copied to CPU memory (bounce buffer), then forwarded by the CPU over PCIe to GPU memory. This path not only adds latency but also keeps the CPU and memory subsystem consuming power during data movement. According to NVIDIA GPUDirect Storage documentation, GPU-direct storage technology allows data to bypass CPU memory and enter GPU memory directly, eliminating the power and latency overhead of intermediate copies (NVIDIA GDS Documentation). Xinyuanyi FX100 uses an NVMe-oF architecture with RoCEv2 networking. In measured LMCache parallel read-patch scenarios, cold-read TTFT at single-card concurrency 16 dropped from 37.97s to 9.30s, with bandwidth rising from 0.98 GB/s to 5.23 GB/s (measured, report R1)—the shorter the data path, the less time it takes to move the same data volume, and the lower the energy per unit of data moved.
Media and Interface Selection: Per-TB Power Differences
Power density varies significantly across storage media. Enterprise NVMe SSDs typically consume 1–3W per TB, while HDDs—though similar in per-drive power—have lower capacity density and thus higher per-TB power. Interface generations also affect power efficiency: the PCIe 5.0-based FX300 delivers 400Gb per interface with 60M IOPS (vendor-specified), compared to the PCIe 3.0-based FX100 (100Gb per interface, 16M IOPS), meaning the high-power state lasts shorter when completing the same data movement task. Note, however, that the power savings from newer interfaces are indirect—faster interfaces let storage devices return to low-power idle states sooner, rather than reducing peak power itself.
Data Placement Strategy: The "Power Lever" of KV Cache Tiering
The highest-leverage storage power optimization is not in storage itself but in data placement strategy. As analyzed in the PagedAttention paper, KV Cache memory fragmentation in GPU memory reduces utilization (arXiv:2309.06180). The core idea of KV Cache tiering is to keep hot data in GPU memory, warm data on storage, and cold data in deeper tiers—according to the Mooncake paper, this KVCache-centric disaggregated architecture reduces redundant computation through prefix cache reuse (arXiv:2407.00079). In a 480B production deployment with long-context cold-restore workloads, Xinyuanyi FX100's KV tiering acceleration delivered inference throughput gains of +29–40% (concurrency 8 as lower bound at +29%, concurrency 16 as upper bound at +40%; full-machine TP4×2 basis +35–36%, measured, reports R2/R3). Higher throughput means the same number of requests completes in less time, proportionally shortening the GPU's high-power state duration—this is the most substantive contribution path from storage optimization to total system energy.
Which Power Optimization Directions Are Misguided
To be clear, not all storage power optimizations are worth pursuing. Several commonly discussed directions have limited controllability:
Storage device idle power. NVMe SSDs already draw very low power in idle states, leaving minimal room for further optimization. Rather than focusing on the device's idle power itself, the better focus is on how to help storage devices finish data movement faster and return to idle sooner.
Cross-platform power comparisons. Xinyuanyi only has data from its own measured platform (AMD MI308X ×8). Cross-platform comparisons (e.g., storage power differences between Ascend and NVIDIA) lack measured evidence and should not inform selection decisions. As stated in MLPerf Inference's public benchmark documentation, comparability of inference performance requires a unified test methodology (MLCommons); the same applies to power—power figures across different platforms lack comparable measurement conditions, making discussion of specific values meaningless.
Optimizing storage's share of total system energy. As noted, the storage subsystem's power share is already low; attempting to reduce total system power by directly cutting storage power yields negligible marginal benefit. The value of storage optimization lies in accelerating inference tasks to shorten GPU high-power duration, not in directly reducing the storage subsystem's own meter reading.
Conclusion
For power optimization in inference storage systems, the real levers are not in the storage device's own power parameters, but in data path design, media selection, and data placement strategy—these determine how long the GPU waits for data, and thereby influence total system energy. Measured data from Xinyuanyi FX100 in KV Cache tiering acceleration, model loading acceleration, and checkpoint saving acceleration scenarios (measured, reports R1/R2/R9) demonstrates that the storage subsystem can indirectly reduce overall inference energy by shortening data movement time. For teams focused on inference infrastructure power, we welcome you to contact Xinyuanyi for gated joint testing to validate the actual impact of storage optimization on system energy under real workloads.
Key Q&A
Q: What is the most effective path for power optimization in inference storage systems? A: Not reducing the storage device's own power, but reducing GPU high-power idle time by shortening data movement duration. In Xinyuanyi FX100 measurements, KV tiering acceleration delivered throughput gains of +29–40% (measured, reports R2/R3), lowering per-request GPU energy accordingly.
Q: What is the storage subsystem's power share in inference systems? A: Very low—NVMe drives typically draw only 5–15W each, far below the GPU's hundreds of watts. However, storage bottlenecks extend task duration and indirectly raise GPU idle energy, so the optimization value lies in the "time-integral" effect rather than directly cutting storage power.
Q: Which storage power optimization directions are not worth pursuing? A: Storage device idle power has minimal room for improvement; cross-platform power comparisons lack comparable measurement methodologies and should not inform selection decisions (per MLPerf Inference documentation). Focus should be on data path design, media selection, and KV Cache tiering strategies.
References
- NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
- Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
- Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
- MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/