What Spot GPU Pricing Actually Looks Like

Cloud providers have GPU capacity sitting unused between other customers’ reservations. Rather than let it idle, they auction it off at a steep discount, sometimes 60-90% below on-demand. That is spot, and it is one of the three purchase options every cloud GPU provider offers.

Spot

The catch: the provider can take the instance back with little notice if someone with a higher-priority reservation needs it. You get a much lower price in exchange for accepting that your job might get interrupted. The price works like an auction. It floats with supply and demand for that GPU type in that region at that moment, so spot prices move by the hour and sometimes by the minute.

The scary word is “preemption.” A stateful, long-running training job with no checkpointing will lose real work if it gets preempted at hour 40. But for a huge share of AI workloads the risk is manageable and the discount is worth it:

• Batch inference jobs that can be split into small chunks and retried
• Training runs with regular checkpointing, where a preemption costs you minutes, not hours
• Anything with automatic failover to a different provider or instance when one disappears

Spot is a bad idea where there is no tolerance for interruption at all, such as a live user-facing endpoint with strict latency guarantees. That belongs on-demand, or on a mix where spot handles the bulk of the load and on-demand is the fallback.

On-demand and reserved

On-demand: you ask for a GPU, you get it, and you pay the listed hourly rate until you stop it. No commitment, no auction, no risk of losing the instance mid-job. It is the most expensive way to rent a GPU by a wide margin. Use it when an interruption costs you more than the price premium does: a production inference endpoint that cannot go down, or a training run with a hard deadline.

Reserved: you commit to a GPU for a fixed term, usually a year or more, in exchange for a meaningfully lower rate than on-demand. If you know you will need a certain amount of capacity for the next year, this is the cheapest way to get it, assuming you use what you reserved. You pay for it whether you use it or not, so it makes sense for a steady, predictable baseline and none for a workload that spikes and idles.

Most real workloads mix all three. A production system might run its steady floor on reserved capacity, burst onto spot for batch and training work, and fall back to on-demand when spot is unavailable and the job cannot wait.

I will not quote you a spot price here. Spot pricing moves by the hour across providers and regions, so any number I put in this post is wrong by the time you read it. That volatility is the thing worth automating around instead of checking manually.

That is the idea behind the GPU side of Smart Inference: live pricing across clouds, per-GPU billing instead of per-VM, and automatic fallback when the spot instance you were using disappears. You still make the reserved-vs-spot-vs-on-demand call, the system just handles picking the cheapest healthy option within whichever tier you pick.