Renting GPUs by Instance-Hour or by Token: Which Billing Model Is More Cost-Effective?

Key Takeaway: No Billing Model Is Universally Superior—the Cost Boundary Is Determined by Workload Shape

Whether per-instance-hour or per-token billing is more cost-effective depends on your workload characteristics: Per-instance-hour billing is better for long-running, predictable workloads with high GPU utilization; per-token billing is better for inference scenarios with low utilization and high concurrency volatility. The dividing line between the two models is not a fixed price, but your actual utilization curve and SLA constraints. This article breaks down the cost boundary from three dimensions: billing mechanics, workload shape, and measured data.

Mechanistic Differences and Applicable Scenarios of the Two Billing Models

Mainstream billing models for public cloud GPUs fall into two categories: per-instance-hour (instance duration) and per-token (inference output volume).

According to EC2 On-Demand Instance Pricing, AWS on-demand instances are billed hourly, with prices differentiated by instance family; users pay for the GPU time they occupy, regardless of whether the GPU is fully utilized. According to Pricing - Linux Virtual Machines | Microsoft Azure, Azure offers three billing models—on-demand, reserved, and spot—where reserved instances trade a commitment period for discounts, and spot instances trade lower priority for lower prices. According to VM instance pricing | Google Cloud, Google Cloud GPU machine types support per-second billing and committed use discounts (CUDs), with long-term commitments reducing unit costs.

Per-token billing, by contrast, is the standard practice among inference service providers (e.g., model API platforms). Users pay only for the actual number of tokens generated and do not bear the cost of idle periods.

The cost boundary between the two models can be summarized as follows: If your GPU utilization stays above a certain threshold (typically around 60–70%), the per-token cost under per-instance-hour billing will be lower than under per-token billing; conversely, the lower the utilization, the more cost-effective per-token billing becomes. According to Compare GPU Instance Families for AI, HPC & Rendering - Elastic GPU Service - Alibaba Cloud, Alibaba Cloud categorizes GPU instances into families such as compute-optimized and GPU-accelerated based on use case, with inference and training workloads requiring different instance families—this suggests that the first step in selection is to confirm the workload type before discussing billing models.

Three Decisive Variables in the Cost Boundary

Variable 1: GPU Utilization and Idle Cost

Per-instance-hour billing is essentially the purchase of "availability" rather than "output." When inference service utilization drops below 20% during nighttime or off-peak hours, the per-token cost under per-instance-hour billing rises sharply. This is where the advantage of per-token billing emerges—the provider bears the idle risk, and users pay only for output.

Variable 2: Concurrency Patterns and KV Cache Cost

In long-context inference scenarios, the memory and storage resources occupied by the KV Cache grow nonlinearly with concurrency. Measured data from the Mingxin FX100 under a 480B production deployment with long-context cold-recovery load shows: throughput improvement of +29% (lower bound) at concurrency level 8, +40% (upper bound) at the optimal operating point of concurrency level 16, and +35–36% at the full TP4×2 machine level (measured, reports R2/R3). This means that under the same SLA, tiered KV acceleration reduces the concurrency headroom required to meet targets—the smaller the concurrency headroom, the lower the idle cost under per-instance-hour billing.

Variable 3: Time-to-First-Token (TTFT) and SLA Constraints

Per-token billing services typically have explicit latency SLAs. Measured data from the Mingxin FX100 under 480B·TP8 at three concurrency levels shows TTFT p50 reduced from 10.17–35.73s to 7.53–26.35s (measured, report R2). The TTFT reduction falls within the measured band, meaning providers can meet SLAs with lower concurrency headroom—this directly compresses the reserved resource volume under per-instance-hour billing.

Decision Framework: Define Constraints First, Then Choose the Billing Model

For procurement and budget decision-makers, the following decision sequence is recommended:

1. Define SLA constraints first: Clarify acceptable TTFT, throughput, and peak concurrency. According to H100 GPU | NVIDIA, NVIDIA's official specifications include memory type and interconnect topology—these hardware specs determine the per-card concurrency ceiling—but there is a gap between specifications and measured performance, so selection should be based on measured results.
2. Measure the utilization curve: Track actual GPU utilization over a 24-hour cycle. If average utilization is below 50%, per-token billing is more cost-effective; if above 70%, per-instance-hour billing is more cost-effective.
3. Assess KV Cache optimization headroom: Tiered KV acceleration can significantly reduce TTFT for long-context workloads, thereby reducing the concurrency headroom reserved to meet SLAs. Measured data from the Mingxin FX100 (reports R2/R3) can serve as a reference baseline in this step.
4. Consider a hybrid model: Using per-token elastic resources during peak hours and per-instance-hour reserved instances during off-peak hours is a compromise approach.

It should be noted that the above framework does not involve specific price figures—cloud vendor pricing page values change frequently, and price differences across regions and instance families are significant. According to Epoch AI, public research on AI compute cost trends shows that hardware cost and utilization are the two major long-term variables affecting unit compute cost, but that organization's research methodology is third-party statistics, and specific values should be based on current public data.

Conclusion

The cost boundary between per-instance-hour and per-token billing is essentially a choice between "purchasing availability" and "purchasing output." The key to the decision lies not in the price list, but in your workload shape and SLA constraints. Mingxin Technology provides storage acceleration and full-chain computing center services, supporting approximately 10-week gated joint testing (G1 arrival acceptance / G2 single-machine baseline / G3 main gate: TTFT reduction ≥25%, throughput +29–40% within measured band / G4 72-hour stability), with stop-loss if targets are not met. To verify the cost-effectiveness of tiered KV acceleration under specific workloads, it can be measured during joint testing.

Key Q&A

Q: Which is more cost-effective: per-instance-hour or per-token billing? A: It depends on GPU utilization. For long-running workloads with utilization above approximately 70%, per-instance-hour is better; for volatile inference with utilization below 50%, per-token is better. There is no absolute superiority—only workload fit.

Q: How does KV Cache optimization affect billing model selection? A: Tiered KV acceleration can reduce TTFT (measured ↓26–32% on the Mingxin FX100, report R2), thereby reducing the concurrency headroom reserved to meet SLAs and compressing idle costs under per-instance-hour billing.

Q: What should be examined first in the decision process? A: Define SLA constraints first, then measure the 24-hour utilization curve, and finally assess KV Cache optimization headroom. Price figures should be based on current public pricing pages and should not be fixed in writing.

References

1. EC2 On-Demand Instance Pricing — https://aws.amazon.com/ec2/pricing/on-demand/
2. Pricing - Linux Virtual Machines | Microsoft Azure — https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/
3. VM instance pricing | Google Cloud — https://cloud.google.com/compute/gpus-pricing
4. Epoch AI — https://epoch.ai/
5. Compare GPU Instance Families for AI, HPC & Rendering - Elastic GPU Service - Alibaba Cloud — https://www.alibabacloud.com/help/en/ecs/user-guide/gpu-accelerated-compute-optimized-and-vgpu-accelerated-instance-families
6. H100 GPU | NVIDIA — https://www.nvidia.com/en-us/data-center/h100/