Skip to content

Guide library

Guide · Hosting & inference

Spot vs Reserved GPUs for LLM Inference: When to Commit

Spot GPUs are up to 90% cheaper but can be reclaimed with seconds to minutes of warning, while reservations and commitments trade flexibility for certainty. Here is how to split an LLM inference fleet across them.

Metarix Editorial · Nov 9, 2026 · Updated Nov 11, 2026 · 10 min read

Google Cloud logo on the official Google Cloud social card

For LLM inference, use spot GPUs only for work that can be retried, such as batch jobs, evals and embedding backfills, or for extra replicas above a floor that is already safe. Put the traffic you can't drop on capacity you control: on-demand to start, a short-term reservation for a known spike, and a one- or three-year commitment only once your baseline usage has stayed steady for months.

The discounts are real. AWS, Google Cloud and Azure all advertise spot savings of up to 90% or more. But an inference server is not a batch job: it takes time to load model weights, it holds open streams, and it can lose every GPU of one type at once. Here is how each pricing option works, what the warning windows really allow, and a simple way to split your fleet.

The four ways to pay for an inference GPU

Every major cloud sells GPU time in roughly four ways. The labels differ, but the trade-off is the same: you give up flexibility, or accept the risk of losing the machine, in return for a lower price or a capacity guarantee.

Scroll the table horizontally on smaller screens.

GPU purchasing options for inference
OptionWhat you getWhat you give upBest inference fit
On-demandPay per second or hour, stop any timeHighest list price; launches can fail when a GPU type is scarceEarly products, unknown traffic, the safe floor
Spot / preemptibleSpare capacity at a steep discountCan be reclaimed with seconds to minutes of warning; no SLA on Google CloudBatch inference, evals, backfills, burst replicas
Short-term reservationGuaranteed capacity for a set window (days to weeks)Pay whether you use it or not; AWS Capacity Blocks can't be cancelledLaunches, planned spikes, big one-off jobs
1- or 3-year commitmentThe lowest steady rateLock-in to an hourly spend or a resource type for the whole termA baseline that has been stable for months
Summary by Metarix from AWS, Google Cloud and Microsoft documentation, accessed Nov 9, 2026.

Note that a commitment and a capacity guarantee are not the same thing. AWS Savings Plans and Google's committed use discounts lower your rate; they don't by themselves promise a GPU will be free when you need it. Reservations do. On AWS the two stack: Savings Plans and regional Reserved Instance discounts apply automatically to matching On-Demand Capacity Reservations, according to the Capacity Reservations billing docs.

What spot actually gives you on each cloud

Spot terms differ more than the headline discounts suggest. The warning window matters most, because it decides whether a server can drain gracefully or simply disappears. Discounts as of November 9, 2026, from each provider's official pricing page.

Scroll the table horizontally on smaller screens.

Spot GPU terms by provider (as of Nov 9, 2026)
TermAWS EC2 SpotGoogle Cloud Spot VMsAzure Spot VMs
Advertised discountUp to 90% vs On-DemandUp to 91% vs on-demandUp to 90% vs pay-as-you-go
Warning before reclaim2 minutes (best effort)0 s by default, or 120 s if you set a preemption notice durationUp to 30 s via Scheduled Events (best effort)
How you detect itEventBridge event or instance metadata (instance-action)Metadata value preempted turns TRUEScheduled Events endpoint in the VM
Price behaviourSpot price per instance type and regionCan change up to once a dayCan rise above your max price, which evicts the VM
Worth knowingAWS recommends polling the notice every 5 secondsNo SLA; not billed if preempted in the first minute; GPU Spot VMs are preempted during maintenanceEviction can be by capacity or price; evicted VMs aren't restarted automatically
Sources: AWS Spot interruption notices and Spot pricing pages; Google Cloud Spot VMs docs and pricing page; Azure Spot VMs docs and product page. All accessed Nov 9, 2026. Discounts are provider maximums, not typical GPU savings.

Two caveats. First, "up to" means the best case for some instance type somewhere; GPU types in short supply often have much smaller spot discounts and higher interruption rates. Check current rates on the AWS Spot pricing page, Google Cloud Spot VMs pricing and the Azure Spot VMs page before you plan around a number. Second, Google says spot preemption rates vary by zone, time and resource, and AWS describes its notices as best effort. Design as if the warning might not come.

Why inference is harder on spot than batch work

Batch jobs handle preemption well because work is checkpointed and retried. A live inference server has three properties that make interruptions more painful.

Cold starts can be longer than the warning

A new replica has to start the VM, pull a container image, load model weights into GPU memory and warm up before it can serve. For larger models that can take minutes. If your replacement takes longer to become ready than the 30-second or two-minute notice, you lose capacity for a while on every interruption. Measure your own time-to-ready from a cold VM; that number, not the discount, decides how much spot your serving tier can take.

In-flight requests and caches are lost

A reclaimed GPU drops every streaming response it was serving, along with its prompt and KV cache. Short chat completions can be retried by the client. Long agent turns or generations that take a minute or more may not finish inside the warning window at all, so the user sees an error or a restarted answer. Our latency and throughput explainer covers why long outputs dominate end-to-end time.

Interruptions arrive together

Spot capacity for one GPU type in one zone tends to vanish at the same moment, because the provider needs it back for the same reason. Five spot replicas of the same instance type are not five independent chances. Spreading across instance types, zones or regions helps, and Google notes that managed instance groups with instance flexibility can pick machine types with lower observed preemption rates.

Reserving capacity without a multi-year contract

The middle option has improved a lot, and it is often the right answer for inference spikes you can see coming, such as a product launch or a seasonal peak.

  • AWS On-Demand Capacity Reservations hold capacity in one Availability Zone for any duration. They are billed at the equivalent On-Demand rate whether you run instances in them or not, per second with a 60-second minimum (billing docs).
  • AWS Capacity Blocks for ML reserve GPU or Trainium instances for a future window you choose, up to eight weeks ahead, with up to 64 instances per block. You pay up front, cancellations aren't allowed, and blocks end at 11:30 AM UTC, with termination starting at 11:00 AM UTC on the last day (Capacity Blocks docs, pricing). See our Capacity Blocks page.
  • Google Cloud future reservations in calendar mode reserve GPU or TPU VMs from a start date for up to 90 days. For GPUs you can view future availability up to 60 days ahead, and Google explicitly lists short-term expected increases in inference workloads as a use case (calendar mode docs).

The catch with all three is the same: you pay for reserved hours you don't use. A reservation is insurance against a failed launch at the worst moment, not a discount. Size it to the floor you must have during the window, then burst above it with on-demand or spot.

When a one- or three-year commitment pays off

Commitments trade flexibility for rate. AWS Savings Plans commit you to a dollar-per-hour spend for one or three years; AWS says Compute Savings Plans cut costs by up to 66% and apply across instance families, sizes and regions, while EC2 Instance Savings Plans go up to 72% but are tied to one instance family in one region. Google's resource-based committed use discounts cover GPUs, among other resources, for one or three years, with a bigger discount on the three-year plan. Azure Reservations offer one- and three-year terms for virtual machines.

The break-even is simple arithmetic. If a commitment saves a fraction d of the on-demand rate, it beats paying on-demand only when you would otherwise run that capacity for more than (1 − d) of the hours in the term. The table below is illustrative math, not provider pricing:

Scroll the table horizontally on smaller screens.

Illustrative break-even utilization for a commitment
Commitment discountYou must otherwise use the capacity at leastRough meaning
20%80% of hoursOnly for a truly always-on baseline
40%60% of hoursA busy daytime service that idles overnight is borderline
60%40% of hoursPays off even with modest utilization, if the term doesn't outlast your need
Calculated by Metarix. Real discounts depend on provider, GPU type, term and payment option; check the official pricing pages.

The bigger risk for inference is not utilization but obsolescence. A three-year commitment outlives most model and GPU choices. If your model gets smaller, moves to a newer accelerator, or moves to a hosted API, the commitment keeps billing. Flexible plans that follow your spend across instance families, such as AWS Compute Savings Plans, are safer for inference than ones tied to a single instance family.

A practical way to split your inference fleet

Rather than choosing one model, split traffic into layers and pay for each layer differently.

Scroll the table horizontally on smaller screens.

Matching inference workloads to GPU pricing
WorkloadRecommended capacityWhy
Steady baseline (the lowest traffic you see most hours)On-demand now; a flexible commitment once stable for monthsAlways on, so a discount on it is pure savings
Known spikes (launches, events, seasonal peaks)Short-term reservation sized to the must-have floorA failed GPU launch at peak costs more than idle reserved hours
Unpredictable daily peaksOn-demand plus spot replicas above a safe floorSpot loss only reduces headroom, never takes you offline
Batch inference, evals, embedding backfillsSpot, with checkpoints and retriesInterruptions only delay the job
Latency-insensitive API callsA provider batch API instead of your own GPUsOpenAI's Batch API is 50% cheaper with a 24-hour turnaround
Recommendations by Metarix. OpenAI Batch API terms from OpenAI's Batch guide, accessed Nov 9, 2026.

The rule that matters: never let spot be the only thing serving a request path. Keep enough on-demand or reserved replicas to handle normal traffic on their own, so losing every spot node at once only removes headroom. If you run on Kubernetes, put spot and on-demand nodes in separate node pools, give the serving deployment a floor on the on-demand pool, and allow scale-out onto spot.

Also ask whether you need your own GPUs at all. For many teams, a hosted model API behind a gateway with fallbacks is cheaper and simpler than any GPU purchasing plan; our AI gateways comparison and cost per successful task guide cover that route. If you want a GPU without managing instances or reservations, our serverless GPU inference comparison covers list prices and cold starts. The trade-off is that hosted catalogs change on the provider's schedule, as the Workers AI model retirements showed this month.

Handling a spot interruption gracefully

Whatever cloud you use, the pattern is the same: watch for the notice, stop taking new requests, let short requests finish, and tell clients to retry elsewhere. Here is an illustrative watcher for AWS that polls instance metadata every five seconds, as AWS recommends. It was not run in this session; adapt the drain step to your load balancer and inference server.

#!/usr/bin/env bash
# Illustrative: drain an inference server when AWS posts a Spot interruption notice
while true; do
TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
-H "X-aws-ec2-metadata-token-ttl-seconds: 300")
CODE=$(curl -s -o /dev/null -w '%{http_code}' -H "X-aws-ec2-metadata-token: $TOKEN" \
http://169.254.169.254/latest/meta-data/spot/instance-action)
if [ "$CODE" = "200" ]; then
echo "Spot interruption notice received, draining"
touch /tmp/draining # your readiness probe should fail when this file exists
sleep 90 # let in-flight requests finish inside the 2-minute window
exit 0
fi
sleep 5
done

The metadata path returns 404 until AWS schedules a stop or termination, then returns the action and time, per the AWS interruption notice docs. On Google Cloud, set the preemption notice duration to 120 seconds and wait on the preempted metadata value; on Azure, poll Scheduled Events and expect 30 seconds at most. In every case, make the readiness check fail first so the load balancer stops sending new requests before the GPU goes away.

Verdict

Start on on-demand for anything a user is waiting on, and add spot only as burst capacity above a floor that can carry normal traffic alone. Move batch inference, evals and backfills to spot right away; that is where the discount is easy money. Use short-term reservations, such as AWS Capacity Blocks or Google calendar-mode reservations, for launches and spikes you can see coming.

Commit for one or three years only after your baseline has held steady for several months, and prefer flexible spend-based plans over ones tied to a single GPU family, since your model and hardware will probably change before the term ends. If you are deploying agents on Amazon Web Services, our guide to deploying an AI agent on AWS covers the compute side; for Google Cloud, start with Spot VMs for the batch tier only.

Sources

Tools covered in this guide

Frequently asked questions

Can I run LLM inference on spot instances?
Yes, but only for work that tolerates interruption, such as batch inference and evals, or as extra replicas above a floor of on-demand or reserved capacity. AWS gives about two minutes of warning, Azure up to 30 seconds, and Google Cloud 0 or 120 seconds depending on settings, which is often shorter than the time a new replica needs to load a model.
How much cheaper are spot GPUs than on-demand?
As of November 9, 2026, AWS advertises Spot discounts of up to 90%, Google Cloud up to 91% and Azure up to 90%. These are maximums; popular GPU types often have smaller discounts and higher interruption rates, so check each provider's current spot pricing.
What is the difference between a GPU reservation and a committed use discount?
A reservation guarantees that capacity is available for a period, and you pay for it whether you use it or not. A commitment, such as AWS Savings Plans or Google committed use discounts, lowers your rate in return for one or three years of spend or usage but does not by itself guarantee capacity.
How far ahead can I reserve GPUs for a short period?
AWS Capacity Blocks for ML can start up to eight weeks in the future and can't be cancelled. Google Cloud future reservations in calendar mode reserve GPU VMs for up to 90 days, with GPU availability visible up to 60 days ahead.

More in Hosting & inference

Share