Guide · Hosting & inference
Spot vs Reserved GPUs for LLM Inference: When to Commit
Spot GPUs are up to 90% cheaper but can be reclaimed with seconds to minutes of warning, while reservations and commitments trade flexibility for certainty. Here is how to split an LLM inference fleet across them.
Metarix Editorial · Nov 9, 2026 · Updated Nov 11, 2026 · 10 min read
On this page

For LLM inference, use spot GPUs only for work that can be retried, such as batch jobs, evals and embedding backfills, or for extra replicas above a floor that is already safe. Put the traffic you can't drop on capacity you control: on-demand to start, a short-term reservation for a known spike, and a one- or three-year commitment only once your baseline usage has stayed steady for months.
The discounts are real. AWS, Google Cloud and Azure all advertise spot savings of up to 90% or more. But an inference server is not a batch job: it takes time to load model weights, it holds open streams, and it can lose every GPU of one type at once. Here is how each pricing option works, what the warning windows really allow, and a simple way to split your fleet.
The four ways to pay for an inference GPU
Every major cloud sells GPU time in roughly four ways. The labels differ, but the trade-off is the same: you give up flexibility, or accept the risk of losing the machine, in return for a lower price or a capacity guarantee.
Scroll the table horizontally on smaller screens.
| Option | What you get | What you give up | Best inference fit |
|---|---|---|---|
| On-demand | Pay per second or hour, stop any time | Highest list price; launches can fail when a GPU type is scarce | Early products, unknown traffic, the safe floor |
| Spot / preemptible | Spare capacity at a steep discount | Can be reclaimed with seconds to minutes of warning; no SLA on Google Cloud | Batch inference, evals, backfills, burst replicas |
| Short-term reservation | Guaranteed capacity for a set window (days to weeks) | Pay whether you use it or not; AWS Capacity Blocks can't be cancelled | Launches, planned spikes, big one-off jobs |
| 1- or 3-year commitment | The lowest steady rate | Lock-in to an hourly spend or a resource type for the whole term | A baseline that has been stable for months |
Note that a commitment and a capacity guarantee are not the same thing. AWS Savings Plans and Google's committed use discounts lower your rate; they don't by themselves promise a GPU will be free when you need it. Reservations do. On AWS the two stack: Savings Plans and regional Reserved Instance discounts apply automatically to matching On-Demand Capacity Reservations, according to the Capacity Reservations billing docs.
What spot actually gives you on each cloud
Spot terms differ more than the headline discounts suggest. The warning window matters most, because it decides whether a server can drain gracefully or simply disappears. Discounts as of November 9, 2026, from each provider's official pricing page.
Scroll the table horizontally on smaller screens.
| Term | AWS EC2 Spot | Google Cloud Spot VMs | Azure Spot VMs |
|---|---|---|---|
| Advertised discount | Up to 90% vs On-Demand | Up to 91% vs on-demand | Up to 90% vs pay-as-you-go |
| Warning before reclaim | 2 minutes (best effort) | 0 s by default, or 120 s if you set a preemption notice duration | Up to 30 s via Scheduled Events (best effort) |
| How you detect it | EventBridge event or instance metadata (instance-action) | Metadata value preempted turns TRUE | Scheduled Events endpoint in the VM |
| Price behaviour | Spot price per instance type and region | Can change up to once a day | Can rise above your max price, which evicts the VM |
| Worth knowing | AWS recommends polling the notice every 5 seconds | No SLA; not billed if preempted in the first minute; GPU Spot VMs are preempted during maintenance | Eviction can be by capacity or price; evicted VMs aren't restarted automatically |
Two caveats. First, "up to" means the best case for some instance type somewhere; GPU types in short supply often have much smaller spot discounts and higher interruption rates. Check current rates on the AWS Spot pricing page, Google Cloud Spot VMs pricing and the Azure Spot VMs page before you plan around a number. Second, Google says spot preemption rates vary by zone, time and resource, and AWS describes its notices as best effort. Design as if the warning might not come.
Why inference is harder on spot than batch work
Batch jobs handle preemption well because work is checkpointed and retried. A live inference server has three properties that make interruptions more painful.
Cold starts can be longer than the warning
A new replica has to start the VM, pull a container image, load model weights into GPU memory and warm up before it can serve. For larger models that can take minutes. If your replacement takes longer to become ready than the 30-second or two-minute notice, you lose capacity for a while on every interruption. Measure your own time-to-ready from a cold VM; that number, not the discount, decides how much spot your serving tier can take.
In-flight requests and caches are lost
A reclaimed GPU drops every streaming response it was serving, along with its prompt and KV cache. Short chat completions can be retried by the client. Long agent turns or generations that take a minute or more may not finish inside the warning window at all, so the user sees an error or a restarted answer. Our latency and throughput explainer covers why long outputs dominate end-to-end time.
Interruptions arrive together
Spot capacity for one GPU type in one zone tends to vanish at the same moment, because the provider needs it back for the same reason. Five spot replicas of the same instance type are not five independent chances. Spreading across instance types, zones or regions helps, and Google notes that managed instance groups with instance flexibility can pick machine types with lower observed preemption rates.
Reserving capacity without a multi-year contract
The middle option has improved a lot, and it is often the right answer for inference spikes you can see coming, such as a product launch or a seasonal peak.
- AWS On-Demand Capacity Reservations hold capacity in one Availability Zone for any duration. They are billed at the equivalent On-Demand rate whether you run instances in them or not, per second with a 60-second minimum (billing docs).
- AWS Capacity Blocks for ML reserve GPU or Trainium instances for a future window you choose, up to eight weeks ahead, with up to 64 instances per block. You pay up front, cancellations aren't allowed, and blocks end at 11:30 AM UTC, with termination starting at 11:00 AM UTC on the last day (Capacity Blocks docs, pricing). See our Capacity Blocks page.
- Google Cloud future reservations in calendar mode reserve GPU or TPU VMs from a start date for up to 90 days. For GPUs you can view future availability up to 60 days ahead, and Google explicitly lists short-term expected increases in inference workloads as a use case (calendar mode docs).
The catch with all three is the same: you pay for reserved hours you don't use. A reservation is insurance against a failed launch at the worst moment, not a discount. Size it to the floor you must have during the window, then burst above it with on-demand or spot.
When a one- or three-year commitment pays off
Commitments trade flexibility for rate. AWS Savings Plans commit you to a dollar-per-hour spend for one or three years; AWS says Compute Savings Plans cut costs by up to 66% and apply across instance families, sizes and regions, while EC2 Instance Savings Plans go up to 72% but are tied to one instance family in one region. Google's resource-based committed use discounts cover GPUs, among other resources, for one or three years, with a bigger discount on the three-year plan. Azure Reservations offer one- and three-year terms for virtual machines.
The break-even is simple arithmetic. If a commitment saves a fraction d of the on-demand rate, it beats paying on-demand only when you would otherwise run that capacity for more than (1 − d) of the hours in the term. The table below is illustrative math, not provider pricing:
Scroll the table horizontally on smaller screens.
| Commitment discount | You must otherwise use the capacity at least | Rough meaning |
|---|---|---|
| 20% | 80% of hours | Only for a truly always-on baseline |
| 40% | 60% of hours | A busy daytime service that idles overnight is borderline |
| 60% | 40% of hours | Pays off even with modest utilization, if the term doesn't outlast your need |
The bigger risk for inference is not utilization but obsolescence. A three-year commitment outlives most model and GPU choices. If your model gets smaller, moves to a newer accelerator, or moves to a hosted API, the commitment keeps billing. Flexible plans that follow your spend across instance families, such as AWS Compute Savings Plans, are safer for inference than ones tied to a single instance family.
A practical way to split your inference fleet
Rather than choosing one model, split traffic into layers and pay for each layer differently.
Scroll the table horizontally on smaller screens.
| Workload | Recommended capacity | Why |
|---|---|---|
| Steady baseline (the lowest traffic you see most hours) | On-demand now; a flexible commitment once stable for months | Always on, so a discount on it is pure savings |
| Known spikes (launches, events, seasonal peaks) | Short-term reservation sized to the must-have floor | A failed GPU launch at peak costs more than idle reserved hours |
| Unpredictable daily peaks | On-demand plus spot replicas above a safe floor | Spot loss only reduces headroom, never takes you offline |
| Batch inference, evals, embedding backfills | Spot, with checkpoints and retries | Interruptions only delay the job |
| Latency-insensitive API calls | A provider batch API instead of your own GPUs | OpenAI's Batch API is 50% cheaper with a 24-hour turnaround |
The rule that matters: never let spot be the only thing serving a request path. Keep enough on-demand or reserved replicas to handle normal traffic on their own, so losing every spot node at once only removes headroom. If you run on Kubernetes, put spot and on-demand nodes in separate node pools, give the serving deployment a floor on the on-demand pool, and allow scale-out onto spot.
Also ask whether you need your own GPUs at all. For many teams, a hosted model API behind a gateway with fallbacks is cheaper and simpler than any GPU purchasing plan; our AI gateways comparison and cost per successful task guide cover that route. If you want a GPU without managing instances or reservations, our serverless GPU inference comparison covers list prices and cold starts. The trade-off is that hosted catalogs change on the provider's schedule, as the Workers AI model retirements showed this month.
Handling a spot interruption gracefully
Whatever cloud you use, the pattern is the same: watch for the notice, stop taking new requests, let short requests finish, and tell clients to retry elsewhere. Here is an illustrative watcher for AWS that polls instance metadata every five seconds, as AWS recommends. It was not run in this session; adapt the drain step to your load balancer and inference server.
#!/usr/bin/env bash# Illustrative: drain an inference server when AWS posts a Spot interruption noticewhile true; do TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \ -H "X-aws-ec2-metadata-token-ttl-seconds: 300") CODE=$(curl -s -o /dev/null -w '%{http_code}' -H "X-aws-ec2-metadata-token: $TOKEN" \ http://169.254.169.254/latest/meta-data/spot/instance-action) if [ "$CODE" = "200" ]; then echo "Spot interruption notice received, draining" touch /tmp/draining # your readiness probe should fail when this file exists sleep 90 # let in-flight requests finish inside the 2-minute window exit 0 fi sleep 5done
The metadata path returns 404 until AWS schedules a stop or termination, then returns the action and time, per the AWS interruption notice docs. On Google Cloud, set the preemption notice duration to 120 seconds and wait on the preempted metadata value; on Azure, poll Scheduled Events and expect 30 seconds at most. In every case, make the readiness check fail first so the load balancer stops sending new requests before the GPU goes away.
Verdict
Start on on-demand for anything a user is waiting on, and add spot only as burst capacity above a floor that can carry normal traffic alone. Move batch inference, evals and backfills to spot right away; that is where the discount is easy money. Use short-term reservations, such as AWS Capacity Blocks or Google calendar-mode reservations, for launches and spikes you can see coming.
Commit for one or three years only after your baseline has held steady for several months, and prefer flexible spend-based plans over ones tied to a single GPU family, since your model and hardware will probably change before the term ends. If you are deploying agents on Amazon Web Services, our guide to deploying an AI agent on AWS covers the compute side; for Google Cloud, start with Spot VMs for the batch tier only.
Sources
- AWS: Spot Instance interruption notices (accessed Nov 9, 2026)
- AWS: Amazon EC2 Spot Instances pricing (accessed Nov 9, 2026)
- AWS: Capacity Blocks for ML and Capacity Blocks pricing (accessed Nov 9, 2026)
- AWS: Capacity Reservations pricing and billing (accessed Nov 9, 2026)
- AWS: Savings Plans pricing (accessed Nov 9, 2026)
- Google Cloud: Spot VMs and Spot VMs pricing (accessed Nov 9, 2026)
- Google Cloud: Committed use discounts overview (accessed Nov 9, 2026)
- Google Cloud: Future reservation requests in calendar mode (accessed Nov 9, 2026)
- Microsoft: Azure Spot Virtual Machines and Azure Spot product page (accessed Nov 9, 2026)
- Microsoft: Save costs with Azure Reservations (accessed Nov 9, 2026)
- OpenAI: Batch API guide (accessed Nov 9, 2026)
Tools covered in this guide
Frequently asked questions
- Can I run LLM inference on spot instances?
- Yes, but only for work that tolerates interruption, such as batch inference and evals, or as extra replicas above a floor of on-demand or reserved capacity. AWS gives about two minutes of warning, Azure up to 30 seconds, and Google Cloud 0 or 120 seconds depending on settings, which is often shorter than the time a new replica needs to load a model.
- How much cheaper are spot GPUs than on-demand?
- As of November 9, 2026, AWS advertises Spot discounts of up to 90%, Google Cloud up to 91% and Azure up to 90%. These are maximums; popular GPU types often have smaller discounts and higher interruption rates, so check each provider's current spot pricing.
- What is the difference between a GPU reservation and a committed use discount?
- A reservation guarantees that capacity is available for a period, and you pay for it whether you use it or not. A commitment, such as AWS Savings Plans or Google committed use discounts, lowers your rate in return for one or three years of spend or usage but does not by itself guarantee capacity.
- How far ahead can I reserve GPUs for a short period?
- AWS Capacity Blocks for ML can start up to eight weeks in the future and can't be cancelled. Google Cloud future reservations in calendar mode reserve GPU VMs for up to 90 days, with GPU availability visible up to 60 days ahead.