$/M Token Calculation for AI Compute Centers

The $/M token (cost per million tokens of inference) at an AI compute center is not a fixed industry quote but an engineering variable determined by hardware acquisition, power consumption, depreciation cycles, utilization, and effective throughput. This article presents a reproducible measurement framework: first define the components of total cost of ownership (TCO), then divide by the total deliverable tokens over the lifecycle, and finally identify cost-reduction levers through sensitivity analysis of key variables. This methodology can be applied directly to pre-investment calculations and operational cost benchmarking.

1. Basic Structure of the Cost Model: TCO Divided by Effective Output

The mathematical definition of $/M token is straightforward:

$/M token = Lifecycle TCO ÷ Lifecycle effective token output

However, both sub-terms contain details that are easily underestimated. TCO is not the purchase price but includes the following five components:

1. Hardware acquisition cost: GPU servers, storage arrays, network equipment (switches, optical modules), racks, and cabling. Based on the reference price of ¥371,200 for a fully configured Mingxin FX100 (approximately ¥2,014/TB), the acquisition cost of a single storage device typically accounts for 5–15% of total hardware investment in a compute center, depending on the GPU-to-storage ratio.
2. Power and cooling infrastructure: Includes UPS, PDU, precision air conditioning, and chillers. This Capex typically accounts for 20–30% of total data center investment and is strongly correlated with GPU power consumption.
3. Operations and labor: 7×24 shift coverage, hardware replacement, firmware upgrades. Per industry practice, annual operations cost is approximately 3–8% of hardware acquisition cost.
4. Power consumption (dominant Opex): GPU full-load power × load factor × electricity price × operating hours. Using an 8× AMD MI308X platform as an example, single-machine power is approximately 6–8 kW; with a PUE of 1.3, actual consumption is multiplied by 1.3.
5. Depreciation and cost of capital: GPU servers are typically depreciated over 3–5 years; storage and network equipment over 5–7 years. Cost of capital is calculated based on loan interest rates or opportunity cost.

The "effective token output" in the denominator also requires precise definition. It is not the theoretical peak token count, but rather:

Effective output = Available hours × Average utilization × Effective throughput (token/s) × Time conversion

Among these, "average utilization" is the largest implicit discount factor. GPU utilization in production environments rarely exceeds 60–70%, due to request queuing, batch processing gaps, cold-start loading, and fault recovery. "Effective throughput" must use measured values under production workloads, not vendor-quoted peak specifications. For example, in measured results from Mingxin report R2, for a 480B model with TP8 across three concurrency levels, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s (source: measured, report R2), with throughput gains translating directly into lower unit costs.

2. Deriving Unit Cost from Measured Data: A Reproducible Example

The following simplified example demonstrates the complete derivation path. Assume a compute center deploys an 8-card MI308X platform targeting long-context inference for a 480B MoE model.

Step 1: Determine annualized TCO

• Hardware acquisition: 8 × MI308X (approximately ¥1.5 million/card) + 2 × EPYC 9654 servers + storage (FX100 fully configured at ¥371,200) + networking ≈ ¥15 million
• Power infrastructure allocation: 25% of hardware ≈ ¥3.75 million
• Annual operations (5% of hardware) ≈ ¥750,000
• Annual power: 7 kW per machine × 24h × 365 days × 1.3 PUE × ¥0.8/kWh ≈ ¥637,000
• 3-year depreciation + 5% cost of capital, annualized TCO ≈ (15,000,000 + 3,750,000)/3 + 750,000 + 637,000 ≈ ¥7.637 million/year

Step 2: Determine annual effective output

• Available hours: 365 × 24 × 0.95 (5% for faults and maintenance) = 8,322 hours
• Average utilization: 60% (conservative production value)
• Effective throughput: Based on measured results from Mingxin report R2, at 480B·TP8 with 8-way concurrency and no external recomputation, throughput is approximately 74.9 tok/s (vs. 4.1 tok/s recomputation baseline, source: measured, report R2). Conservatively take 50 tok/s (accounting for batch processing gaps)
• Annual effective output = 8,322h × 3600s × 0.6 × 50 tok/s ≈ 8.99 × 10^11 tokens

Step 3: Calculate $/M token

¥7.637 million / 8.99 × 10^11 tokens × 10^6 ≈ ¥8.5 / M token (approximately $1.2 / M token at an exchange rate of 7.2)

This result is comparable to the publicly reported inference cost range of $0.5–3 / M token, validating the reasonableness of the model framework. It should be emphasized that this figure is highly sensitive to assumptions—if utilization drops from 60% to 40%, unit cost rises by 50%; if throughput increases from 50 to 74.9 tok/s (upper bound from Mingxin report R2 measurements), unit cost falls by 33%.

3. Sensitivity of Key Variables: What Affects $/M token Most

Single-variable sensitivity analysis of the above model yields the following cost-reduction priority ranking:

1. Effective throughput (greatest impact): For every 20% increase in throughput, $/M token decreases by approximately 16.7%. Improvement paths include: KV Cache offloading to reduce recomputation, better batch scheduling, and faster model loading. Mingxin report R2 measurements show that tiered KV acceleration delivers +29–40% throughput gains in production 480B deployments (source: measured, reports R2/R3), equivalent to a direct unit cost reduction of 22–29%. The acceleration factor over no-external-recomputation baselines reaches 8.6–20× (source: measured, report R2), meaning that in long-context scenarios, storage performance constrains throughput far more than intuition suggests.

2. Utilization: Increasing utilization from 50% to 70% reduces $/M token by 28.6%. However, utilization is constrained by business load fluctuations and is difficult to improve substantially in the short term. A more practical path is reducing cold-start time—Mingxin report R9 measurements show model inference loading accelerated by 6.2–9.3× (vs. NFS baseline, source: measured, report R9), compressing service scaling time from 23 minutes to 2.5 minutes, indirectly improving available hours.

3. Power cost: For every ¥0.1/kWh decrease in electricity price, $/M token drops by approximately 1.5%. Siting in regions with abundant hydroelectric or green power can yield a 5–10% cost advantage. However, this is a siting decision, not an operational optimization.

4. Depreciation period: Extending GPU depreciation from 3 to 4 years reduces $/M token by approximately 8%. This must be weighed against performance obsolescence risk—the energy efficiency improvement of new-generation GPUs typically exceeds 30% per year.

Conclusion: Given fixed hardware selection, improving effective throughput is the most controllable and quantifiable cost-reduction lever for $/M token. This is why storage performance (KV Cache offloading, Checkpoint saving) is increasingly important in inference cost models—it directly determines the proportion of GPU idle time.

4. From Single Machine to Compute Center: Cost Amortization at Scale

The above example takes a single-machine perspective. At the compute center level, $/M token must also incorporate two scaling factors:

1. Scale amortization: Data center infrastructure (power, cooling, security) exhibits step-function costs below 100 cards, with marginal costs declining significantly beyond 500 cards. Industry experience: at scales above 500 cards, infrastructure unit costs decrease by 30–40%.

2. Mixed workload optimization: Temporal complementarity between inference and training loads can improve overall utilization. Training tasks typically run in nightly batches, while inference workloads dominate daytime hours. Mingxin report R1 measurements show training Checkpoint saving accelerated by 1.9× (178s → 94s, source: measured, report R1), meaning reduced GPU interruption time for training tasks, freeing more time slots for inference workloads.

Compute center decision-makers should establish the correct understanding: $/M token is not a procurement metric but an operational metric. Hardware procurement only sets the cost floor; operational efficiency (throughput, utilization, cold-start time) determines the gap between actual cost and that floor. We recommend recalculating quarterly using this framework, quantifying changes in throughput and utilization into $/M token movements to drive continuous optimization.

Conclusion

A reproducible cost model is the infrastructure for compute build-out decisions. Mingxin Technology provides storage acceleration and full-chain compute center services, supporting gated joint testing (G1–G4) to verify the actual impact of storage performance on inference throughput. The measurement model can be reproduced in Python after NDA execution. Teams with compute cost optimization needs are welcome to contact us for joint testing collaboration.

Key Q&A

Q: What is the core formula for calculating $/M token? A: $/M token = Lifecycle TCO ÷ Lifecycle effective token output. TCO includes hardware, power infrastructure, operations, electricity, and depreciation; effective output = available hours × utilization × effective throughput.

Q: Which variable has the greatest impact on $/M token? A: Effective throughput. For every 20% increase in throughput, unit cost drops by approximately 16.7%. Mingxin report R2 measurements show tiered KV acceleration delivers +29–40% throughput gains (source: measured, reports R2/R3), making it the most quantifiable cost-reduction lever currently available.

Q: How can the actual impact of storage performance on inference cost be verified? A: Through gated joint testing: the G3 primary gate requires TTFT reduction ≥25% and throughput +29–40% measured in-band; failure to meet these thresholds triggers early termination. The measurement model can be reproduced in Python after NDA execution.