What Spot GPU Pricing Actually Looks Like

Every cloud GPU provider offers roughly the same three purchase options, and roughly nobody explains them clearly. So here is the plain version.

On-demand

You ask for a GPU, you get it, you pay the listed hourly rate until you stop it. No commitment, no auction, no risk of losing the instance mid-job. This is the “just give me the thing” option, and it is priced accordingly, it is the most expensive way to rent a GPU by a wide margin.

Use on-demand when you need a guarantee. A production inference endpoint that cannot go down, a training run with a hard deadline, anything where an interruption costs you more than the price premium does.

Reserved

You commit to a GPU for a fixed term, usually a year or more, in exchange for a meaningfully lower rate than on-demand. You are trading flexibility for a discount. If you know with confidence that you will need a certain amount of GPU capacity for the next year, this is the cheapest way to get it, assuming you actually use what you reserved.

The catch is obvious once you say it out loud: you are paying for capacity whether you use it or not. Reserved instances make sense for a steady, predictable baseline load. They make no sense for a workload that spikes and idles.

Spot

This is the interesting one. Cloud providers have GPU capacity sitting unused between other customers’ reservations, and rather than let it sit idle, they auction it off at a steep discount, sometimes 60-90% below on-demand. The catch: the provider can take it back with little notice if someone with a higher-priority reservation needs it.

That is the whole trade. You get a much lower price in exchange for accepting that your job might get interrupted. The pricing itself works like an actual auction, it floats based on supply and demand for that GPU type in that region at that moment, which is why spot prices move by the hour and sometimes by the minute.

Preemption risk, and why it matters less than people think

The scary part of spot is the word “preemption.” Your instance can be reclaimed mid-job. For a lot of workloads this sounds disqualifying, and for some it is. A stateful, long-running training job with no checkpointing will lose real work if it gets preempted at hour 40.

But for a huge share of AI workloads, this risk is manageable and the discount is worth it:

• Batch inference jobs that can be split into small chunks and retried
• Training runs with regular checkpointing, where a preemption costs you minutes, not hours
• Anything with automatic failover to a different provider or instance when one disappears

The workloads where spot is a bad idea are the ones with no tolerance for interruption at all, a live user-facing endpoint with strict latency guarantees, for instance. That belongs on-demand, or on a mix where spot handles the bulk of the load and on-demand is the fallback.

When to use which

Rough rule of thumb, not a formula:

• Predictable, steady, 24/7 baseline load → reserved
• Interruption-tolerant, bursty, or batch workloads → spot
• Anything that absolutely cannot fail or wait → on-demand

Most real workloads are a mix of all three, not a single choice. A production system might run its steady floor on reserved capacity, burst onto spot for batch and training work, and fall back to on-demand when spot is unavailable and the job cannot wait.

I will not quote you a spot price here, and that is deliberate. Spot pricing moves by the hour across providers and regions, so any number I put in this post is wrong by the time you read it. That volatility is exactly the thing worth automating around instead of checking manually.

That is the whole idea behind the GPU side of Smart Inference: live pricing across clouds, per-GPU billing instead of per-VM, and automatic fallback when the spot instance you were using disappears. You still make the reserved-vs-spot-vs-on-demand call, the system just handles picking the cheapest healthy option within whichever tier you pick.