What Is a GPU Spot Instance? Explained Simply

A GPU spot instance is spare GPU capacity that a cloud provider sells at a steep discount because it is not currently reserved by another customer. The trade-off is that the provider can reclaim it with little notice if the capacity is needed elsewhere. In exchange for accepting that risk, spot pricing typically runs 50 to 80 percent below the same card’s on-demand rate.

Why spare GPU capacity exists

Cloud providers buy or build GPU fleets sized for their peak demand, not their average demand. Between reservations, during off-hours, and in regions with lighter load, a meaningful share of that hardware sits idle. Idle hardware earns nothing, so providers sell the slack at a discount rather than let it sit unused. It is the same economic logic as an airline selling a standby seat cheap rather than fly with it empty, except the “flight” is an hour of GPU time and the “standby passenger” can be bumped mid-flight if a full-price customer needs the seat.

How spot pricing actually moves

Spot pricing behaves like an auction, not a fixed rate card. It floats with supply and demand for that specific GPU model, in that specific region, at that specific moment. When idle capacity builds up, the price drops. When demand spikes, whether from a training run, a burst of inference traffic, or simply less supply in a region, the price climbs. This is why the same GPU model can show noticeably different prices across providers or regions at the same instant, and why a price you see right now may already be stale within the hour.

The interruption trade-off

The defining risk of a spot instance is preemption: the provider can end your session with a short warning window when it needs the capacity back. How much this matters depends entirely on the workload:

• Good fits for spot: batch inference jobs that can be split into small, retryable chunks; training runs with regular checkpointing, where a preemption costs minutes of lost work, not hours; anything with automatic failover to a different provider or instance when one disappears.
• Poor fits for spot: a live, user-facing endpoint with a strict latency guarantee and zero tolerance for a mid-request interruption; a long, stateful job with no checkpointing, where losing the instance loses real, unrecoverable progress.

Most production systems that use spot capacity do not run entirely on it. A common pattern is a steady baseline on reserved or on-demand capacity, with spot handling bursty or batch work and on-demand as the fallback when spot is unavailable or interruption risk is unacceptable for that particular job.

Spot vs on-demand vs reserved

The three purchase models exist on a spectrum of price versus commitment. On-demand: no commitment, no discount, you get the GPU immediately and pay the full listed rate until you stop it, appropriate when you need a guarantee. Reserved: you commit to a fixed term, usually a year or more, for a meaningfully lower rate than on-demand, appropriate for a steady, predictable baseline load you know you will use. Spot: the deepest discount, with the interruption risk attached, appropriate for anything interruption-tolerant.

Spot instances vs managed inference APIs

It is worth separating “renting a spot GPU” from “calling a hosted inference API,” because they solve overlapping but different problems. A hosted API (including a router that sits in front of several of them) hands you a model behind an endpoint; you never see or manage the underlying hardware, and you pay per token. A spot GPU instance hands you the hardware itself, at an hourly rate; you are responsible for loading the model, serving it, and handling the interruption risk yourself, in exchange for more control over exactly what runs and how. Teams running an unusual model, a fine-tuned checkpoint, or a workload that a hosted API simply does not offer are the ones who end up needing a spot instance directly rather than a token-metered API call.

How AIVory’s spot GPU marketplace works

Smart Inference aggregates live spot offers across RunPod, Vast.ai, AWS Spot, Azure Spot, and Crusoe Cloud into one marketplace, and surfaces the single cheapest live rate for each GPU model rather than making you check each provider separately. Every offer carries an interruption-risk figure so you can see the trade-off before you deploy, not after. GPUs are billed per second after a 15-minute minimum, per-GPU rather than per-VM for multi-GPU rentals, and billed from the same credit balance as Smart Inference’s LLM routing, so renting a GPU and calling a hosted model use the same account. If a model you need is not already hosted, requesting it can trigger an automatic spot GPU deployment rather than requiring you to provision one by hand.

See live spot prices across all 32 tracked GPUs, sorted cheapest first.