AMD MI308X Inference: 192GB HBM Card Performance

AMD MI308X single card with 192GB HBM can fully load a 480B-parameter MoE model (FP8 weights ~450GB) in measured tests, but requires an 8-card distributed deployment. With Mingxin FX100 all-flash NVMe-oF array acceleration, its KV Cache tiered acceleration scheme boosts inference throughput by 29–40%, reduces first-token latency by 26–32%, and significantly alleviates bottlenecks in scenarios without external memory recomputation. The following analysis, based on Mingxin’s measured reports R1–R4, breaks down the practical inference capability boundaries of MI308X within the domestic computing ecosystem.

Actual Load Capacity of a Single 192GB HBM Card

One design goal of the MI308X single card with 192GB HBM is to support memory-intensive loads in large model inference. On an 8-card MI308X cluster, Mingxin tested Qwen3-Coder-480B-FP8 (MoE, weights ~450GB) using TP8 parallelism. In single-card memory allocation, model weights occupy approximately 56GB (450GB / 8), with the remaining space used for KV Cache, activations, and system overhead. Measured results show that under an 8-card configuration, each card can handle approximately 192GB of memory demand, but fully loading the 480B model requires a distributed inference framework (e.g., vLLM 0.20.1+rocm721).

For smaller parameter models, such as DeepSeek-32B or 70B, a single MI308X card can independently run inference services. Mingxin’s tests on the Huawei Atlas 910B platform (measured, report R9) show that with FX100 acceleration, DeepSeek-32B service loading time dropped from 691s to 112s (6.2×), and DeepSeek-70B from 1399s to 150s (9.3×), indicating that single-card memory is sufficient to accommodate such model weights, with the bottleneck primarily in storage I/O rather than memory.

KV Cache Acceleration: Measured Gains in Throughput and Latency

KV Cache is a core bottleneck in long-context inference. On an 8 × MI308X platform, Mingxin used an FX100 all-flash array (4-disk RAID0, 14TB, RoCEv2) as external KV Cache storage, compared against a local NVMe single-disk baseline. At the optimal operating point for the 480B model with TP8 and concurrency 16, the KV tiered acceleration scheme achieved:

• Inference throughput improvement: +40% (from baseline to FX100 acceleration, measured, report R2)
• Time-to-first-token (TTFT) reduction: TTFT p50 decreased from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32% (measured, report R2)
• Acceleration in scenarios without external memory recomputation: Recomputation baseline TTFT p50 of 149.5s (conc16) vs. FX100 at 11.85s, throughput increased from 4.1 to 74.9 tok/s, acceleration factor of 8.6–20× (measured, report R2)

These data indicate that while the MI308X’s 192GB HBM can cache part of the KV Cache, memory overflow is inevitable in long-context scenarios. FX100 offloads KV Cache to an all-flash array via the NVMe-oF protocol, using a tiered strategy (hot data in memory, cold data read on demand) to achieve near-linear acceleration. In LMCache parallel read patch tests (measured, report R1), under single-card concurrency 16 cold read scenarios, TTFT dropped from 37.97s to 9.30s (4.1× improvement), and bandwidth increased from 0.98 to 5.23 GB/s (5.3×).

Comparison with NFS Baseline: Impact of Storage Architecture on Inference Efficiency

Traditional inference deployments often rely on NFS (Network File System) to load models and KV Cache. Mingxin’s comparative tests on the Huawei Atlas 910B platform (measured, report R9) show that the FX100 all-flash array achieves model inference loading acceleration of 6.2–9.3× compared to the NFS baseline. Specifically:

• DeepSeek-32B: Service loading 691s → 112s (6.2×)
• DeepSeek-70B: 1399s → 150s (9.3×)

This gap stems from NFS protocol overhead and the random read/write bottlenecks of mechanical hard drives. FX100, based on the NVMe-oF (RoCEv2) protocol with single-port 100Gb bandwidth, achieves a measured sustained write bandwidth of up to 6.40 GB/s (training checkpoint save scenario, measured, report R1), while NFS on equivalent hardware typically falls below 1 GB/s. For high-compute GPUs like the MI308X, storage I/O becomes a hidden bottleneck for inference latency, especially with frequent KV Cache reads and writes in long-context scenarios.

Conclusion

The AMD MI308X single card with 192GB HBM can support inference of 480B-parameter models in an 8-card cluster, but KV Cache management is the key breakthrough point for memory bottlenecks. Mingxin’s FX100 all-flash array, through the NVMe-oF protocol and tiered acceleration strategy, improves inference throughput by 29–40% in measured tests, reduces first-token latency by 26–32%, and significantly improves response times in scenarios without external memory recomputation. For technical decision-makers in computing centers, if adopting domestic GPU platforms like MI308X or Ascend 910B, it is recommended to prioritize evaluating the impact of storage I/O on inference efficiency and verify the actual benefits of KV Cache acceleration schemes through gated joint testing. Mingxin offers an approximately 10-week joint testing process, supporting Python-reproducible measurement models under NDA to reduce deployment risk.

Key Q&A from This Article

Q: Can a single AMD MI308X card with 192GB HBM independently run a 480B-parameter model? A: No. The 480B model (FP8 weights ~450GB) requires an 8-card distributed deployment; single-card memory is primarily used for weight sharding and KV Cache caching.

Q: What is the inference throughput improvement of Mingxin FX100 on the MI308X platform? A: At the optimal operating point for the 480B model with TP8 and concurrency 16, inference throughput improves by 40% (measured, report R2), with a full-system improvement of 35–36% (measured, report R3).

Q: What advantages does FX100 have over NFS in model loading? A: On the Huawei Atlas 910B platform, DeepSeek-32B loading accelerates by 6.2×, and DeepSeek-70B by 9.3× (measured, report R9), primarily due to the low latency and high bandwidth of the NVMe-oF protocol.