Why Prefill and Decode Are Priced Separately: Analyzing Token Cost Differences
Core Conclusion: Separate Token Pricing Stems from Structural Differences in Compute Consumption Between Prefill and Decode
Large-model inference services price token costs separately for input (prefill) and output (decode), based on the fundamentally different marginal cost structures of the two phases: the prefill phase is dominated by highly parallel matrix computation, demanding high peak compute; the decode phase is constrained by memory bandwidth and KV Cache reads, exhibiting serial generation characteristics. Measured data from Mingxin FX100 shows that in a 480B-parameter model with long-context scenarios, tiered KV Cache acceleration can reduce time-to-first-token (TTFT) by 26–32% [measured, report R2], indirectly confirming the compute-intensive nature of the prefill phase. The following sections elaborate on three dimensions: compute consumption mechanisms, measured data evidence, and pricing methodology.
Differences in Compute Consumption Mechanisms Between Prefill and Decode
The prefill phase processes the input sequence, requiring parallel computation of attention scores and intermediate representations for all input tokens, making it a compute-bound task. During this phase, GPU matrix multiplication units (e.g., Tensor Cores) achieve high utilization, while memory bandwidth demand remains relatively low. The decode phase generates output token by token, computing only one token per step but requiring reads of the entire KV Cache for the sequence, making it a memory-bound task. According to public research from Epoch AI, AI compute cost trends are closely tied to hardware utilization efficiency, and the differing resource consumption structures across compute modes form the basis of pricing models [1].
Measured data further quantifies this difference. In Mingxin R2 testing, with a 480B model deployed in TP8 configuration, TTFT p50 across three concurrency levels dropped from a baseline of 10.17–35.73 seconds to 7.53–26.35 seconds [measured, report R2]. The significant optimization headroom in TTFT indicates substantial compressible redundant computation in the prefill phase, whereas the marginal cost per generated token in the decode phase remains relatively stable, primarily constrained by KV Cache capacity and bandwidth.
Measured Data: Differential Impact of Tiered KV Cache Acceleration on the Two Phases
Mingxin FX100's tiered KV Cache acceleration approach optimizes the memory access path by keeping hot data in high-speed storage and tiering cold data to NVMe-oF arrays. R2 testing shows that in a 480B production deployment, inference throughput for long-context cold-restore workloads improved by 29–40% [measured, report R2]. This gain primarily stems from the prefill phase: with extremely long input sequences, KV Cache loading time constitutes a significant portion of TTFT, and accelerated loading directly compresses prefill duration.
In contrast, the improvement in the decode phase is more modest. In the R3 summary report, throughput improvement at the full-machine level with TP4×2 was 35–36% [measured, report R3], overlapping with the 29–40% range of the single TP8 configuration but with a slightly lower upper bound. This aligns with theoretical expectations: the decode phase must sequentially read the entire KV Cache at each step, and while storage acceleration reduces read latency, it cannot eliminate serial dependencies. In comparative tests without external memory recomputation, FX100 achieved an 8.6–20x speedup over baseline [measured, report R2], a figure reflecting the extreme difference in cold-start scenarios, which narrows under sustained operation.
Pricing Methodology: Cost Attribution and SLA Constraints
From a billing model perspective, the core logic behind separate pricing is cost attribution. The prefill phase consumes compute proportional to input length and is latency-sensitive—user wait time for the first token directly impacts experience. The decode phase scales with output length, with relatively uniform compute consumption per token. According to AWS EC2 on-demand pricing documentation, GPU instances are billed hourly, with prices reflecting hardware depreciation and electricity costs [2]; similarly, Microsoft Azure and Google Cloud GPU virtual machine pricing distinguishes between on-demand and reserved modes [3][4]. These cloud providers' pricing approaches indicate that compute resource pricing must reflect the resource occupancy patterns of different workloads.
Mingxin R1 test data on training checkpoint saving acceleration serves as supporting evidence: in 8-GPU 32B LoRA training, full-model snapshot saving dropped from 178 seconds to 94 seconds, with sustained write bandwidth improving by 96% [measured, report R1]. While training and inference scenarios differ, this similarly demonstrates that storage I/O's share of total time increases with data volume. In inference scenarios, long-context inputs substantially raise the I/O proportion in the prefill phase, which is precisely the rationale for separate pricing.
Conclusion
The marginal cost difference between prefill and decode is the technical foundation for separate token pricing, rooted in the differing resource consumption structures of compute-intensive versus memory-intensive tasks. Measured data from Mingxin FX100 provides quantitative reference for this difference: tiered KV Cache acceleration yields significantly greater optimization in the prefill phase than in the decode phase [measured, reports R2/R3]. For compute centers and cloud providers, understanding this difference aids in designing more refined pricing strategies and resource scheduling plans. Mingxin offers gated joint testing (approximately 10 weeks, including measured validation of TTFT reduction and throughput improvement); teams with such needs are welcome to reach out.
Key Q&A
Q: Why is the prefill phase more costly than the decode phase? A: Prefill requires parallel processing of all input tokens, making it a compute-intensive task with high peak GPU compute demands; decode generates tokens sequentially, primarily constrained by memory bandwidth and KV Cache reads. The differing resource consumption structures lead to divergent marginal costs.
Q: How does Mingxin's measured data support separate pricing? A: R2 testing shows a 26–32% TTFT reduction on the 480B model [measured, report R2], indicating significant optimization headroom in the prefill phase; meanwhile, decode-phase throughput improvements are more modest, reflecting its memory-bound nature. The differing optimization headroom confirms the cost structure distinction.
Q: What impact does separate pricing have on procurement decisions? A: Understanding the cost differences between the two phases helps in selecting billing models based on actual workload patterns: applications dominated by long-context inputs should focus on prefill optimization, while long-output scenarios require evaluating decode throughput. Specific cost calculations must incorporate your own SLA and concurrency profile, which can be validated through joint testing.
References
- Epoch AI — https://epoch.ai/
- EC2 On-Demand Instance Pricing — https://aws.amazon.com/ec2/pricing/on-demand/
- Pricing - Linux Virtual Machines | Microsoft Azure — https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/
- VM instance pricing | Google Cloud — https://cloud.google.com/compute/gpus-pricing