Can You Run a Production LLM on Spot GPUs

Can You Run a Production LLM on Spot GPUs

A spot GPU got reclaimed on me mid-batch-job last year. No warning email, no graceful shutdown window worth mentioning. One second the job was running, the next the instance was gone and I was staring at a half-written output file. That is the whole story of spot GPUs in one sentence: cheap, and it will eventually just take the machine back.

So the real question is not “should I use spot.” It is “which parts of my workload can survive that.”

What Spot Actually Is

Cloud providers sell unused GPU capacity at a steep discount, usually 60-70% cheaper than the on-demand rate for the same card. In exchange, they reserve the right to reclaim it whenever a paying on-demand customer wants that GPU back. AWS calls it Spot, GCP calls it Preemptible or Spot VMs, Azure calls it Spot VMs. Same deal everywhere: cheap and revocable.

The reclaim notice varies by provider. AWS gives you a two-minute warning. GCP gives you 30 seconds. Some of the smaller GPU marketplaces give you nothing at all. If your workload cannot checkpoint and resume inside that window, you lose whatever was in flight.

For training jobs this is a known, solved problem. Checkpoint every N steps, resume from the last checkpoint on a new instance. Annoying, but mechanical. For inference, it is a different animal, because inference does not have a checkpoint. A request either finished or it didn’t.

Where Spot Works Fine

Batch inference is the easy case. If you are running a nightly job that scores 50,000 documents, and each document takes 200ms, a mid-run reclaim just means some documents get retried on the next instance. Build the job so it is idempotent and resumable at the record level, not the run level, and spot pricing is close to free money.

Dev and staging environments are the other easy case. Nobody is going to notice if your staging LLM endpoint has a 30-second gap once a week while a new instance spins up. I run all my own model experimentation on spot. If Fireworks or Novita reclaims the box while I’m poking at a fine-tune, I shrug and requeue it.

Anything with a queue in front of it also tolerates spot well. RabbitMQ, SQS, whatever - if requests sit in a queue and a worker pulls them off, losing a worker just means the queue backs up for a few seconds while a replacement spins up. The requester never sees the reclaim happen.

Where Spot Breaks

Synchronous, latency-critical, user-facing inference is the case that breaks. A customer hits your chatbot endpoint, waits for a token stream, and the GPU underneath that request gets pulled mid-generation. There is no clean way to hand that request to another machine without the user seeing a stall or an error. You cannot checkpoint a half-generated response and resume it on different hardware three seconds later - the KV cache lives on that one GPU and it is gone with the instance.

This is the part people skip when they read “60-70% cheaper” and start moving production traffic over. The discount is real. The reclaim risk is also real, and it does not average out the way training-job interruptions do. One reclaim during a demo call with a paying customer costs you more goodwill than the savings are worth.

The fix is not “don’t use spot for production.” The fix is “don’t use spot as your only capacity for production.”

The Fallback Architecture

The pattern that actually works: route traffic to spot GPUs by default, and fail over to on-demand the instant a spot instance disappears or a provider’s queue depth spikes. Per-second billing on both sides makes this cheap to do - you are not paying for a warm on-demand GPU sitting idle, you are paying for on-demand capacity only during the seconds you are actually failed over to it.

A few things that made this work reliably for me:

  • Bill and provision per GPU, not per VM. A VM often bundles CPU, storage, and network you don’t need for a stateless inference call. Paying per GPU-second keeps the fallback math honest.
  • Watch multiple providers, not one. Spot pricing and availability on Cerebras, Groq, DeepInfra, and Novita move independently. If one pool is thin, another usually isn’t.
  • Keep the failover dumb and fast. A health check that takes 10 seconds to notice a dead instance defeats the purpose. Sub-second failure detection matters more than clever routing logic.
  • Test the failover path on purpose, not by accident. I learned this the hard way (see the opening paragraph). Kill an instance in staging on a schedule and watch what happens to in-flight requests.

I built Smart Inference around exactly this pattern - it tracks spot pricing across providers like Fireworks, Cerebras, Groq, Novita, and DeepInfra, and falls back to on-demand automatically when a spot GPU gets reclaimed. Per-second, per-GPU billing on both sides, so the fallback doesn’t quietly become the expensive default. One honest limitation: it optimizes for cost and availability across providers, not for the absolute lowest possible latency on any single request - if you need guaranteed sub-100ms first-token time no matter what, dedicated on-demand capacity is still the safer call.

The Math That Actually Matters

Do not model spot GPU costs by comparing “spot price” to “on-demand price” and calling it a day. Model it by comparing “blended cost including your fallback rate” to “pure on-demand cost.” If your workload gets reclaimed 5% of the time and you fail over cleanly, you are still paying roughly the spot rate for 95% of your traffic and on-demand for 5%. That blended number is what decides whether spot was worth the engineering effort.

Also budget for the engineering effort honestly. Idempotent request handling, health checks, multi-provider routing - none of that is free to build. If your total inference spend is a few hundred euros a month, the savings might not clear the bar for the complexity you’d add. If you are running meaningful production volume, the math flips fast.

Where I Landed

Batch and async jobs: spot, always, no exceptions. Dev and staging: spot, don’t overthink it. Synchronous production inference: spot with an automatic on-demand fallback, never spot alone. That third bucket is the one worth the setup work, because it is the one where the savings are largest and the failure mode is most visible to actual customers.

The instance that got reclaimed on me last year taught me the expensive way. Cheaper to just architect for it up front.