Comparison · Hosting & inference
Serverless GPU Inference Compared: Pricing and Cold Starts
Modal, Runpod Serverless, Replicate, Baseten and Cloud Run with GPUs compared on list price, cold starts and idle billing, with a worked cost example and a decision flow for picking one.
Metarix Editorial · Nov 11, 2026 · Updated Nov 15, 2026 · 12 min read
Documentation-based research
On this page

Serverless GPU inference means you deploy a model as an endpoint and pay only while a GPU is actually up, with the platform scaling to zero when traffic stops. For bursty workloads that is usually far cheaper than renting a GPU around the clock. But the per-hour rate is only half the bill: cold starts, idle windows and minimum charges decide what you really pay.
This guide compares the main options as of November 11, 2026: Modal, Runpod Serverless, Replicate, Baseten, Google Cloud Run with GPUs, Cloudflare Workers AI and what AWS offers instead. It covers list prices, what each platform bills between requests, a worked cost example and when each one fits.
Quick answer
For custom models with spiky traffic, Modal has the lowest H100 list rate of the specialist platforms and a Python-first workflow, while Runpod Serverless is cheapest on mid-range GPUs and bills the shortest idle window by default (5 seconds). Cloud Run with GPUs is the pick if you already run on Google Cloud and your model fits a 24 GB L4 or a 96 GB RTX PRO 6000. Replicate is best for calling public models, where setup and idle time are free. If a catalog model does the job, skip GPUs entirely and use a per-token API such as Workers AI. And once your GPU is busy most of the day, a dedicated or reserved GPU beats all of them.
Scroll the table horizontally on smaller screens.
| Platform | Billing unit | GPUs offered | Default idle billed | Best for |
|---|---|---|---|---|
| Modal | Per second (GPU, CPU and memory separately) | T4 to B300 | 60 s (2 s to 20 min) | Python teams, fast scale-out, large GPUs |
| Runpod Serverless | Per second, rounded up (all-in GPU rate) | 16 GB cards to B300 | 5 s | Docker control, low mid-range prices |
| Replicate | Per second, or per output on some public models | T4, L40S, A100, H100, H200 | Public: none. Private: a few minutes | Calling public models; hosting with Cog |
| Baseten | Per minute, per replica | T4 to B200 | 900 s scale-down delay | Managed production serving |
| Google Cloud Run | Per second, instance-based, 1 min minimum | L4, RTX PRO 6000 | Idle GPU instances up to 10 min | Teams already on Google Cloud |
| Cloudflare Workers AI | Per token (Neurons) | Cloudflare-managed | None | Catalog models, no GPU management |
| AWS SageMaker | Per instance-hour | EC2 GPU instances | Until scaled in | AWS-only shops (not truly serverless) |
What "serverless GPU" actually means
All of these platforms share the same basic loop. A request arrives, the platform starts a container on a GPU (the cold start), your code loads the model weights into GPU memory, the request runs, and the container waits briefly for more traffic before shutting down. When nothing is running, you pay nothing for compute.
The differences are in the details, and they move the bill more than the headline rate does:
- Billing granularity. Per second (Modal, Runpod, Replicate, Cloud Run) or per minute (Baseten). Cloud Run also charges a one-minute minimum per instance.
- What is bundled. Runpod and Replicate quote one all-in rate per hardware type. Modal and Cloud Run bill the GPU, CPU and memory separately, so the GPU price alone understates the real hourly cost.
- What happens between requests. Every platform except Replicate's public models bills the idle window after the last request, and the defaults range from 5 seconds to 15 minutes.
- Cold start time. A container that boots in a second can still take a minute or more to load a large model. You pay for that load time on every platform that bills setup.
If you are new to the latency side of this, our explainer on LLM latency and throughput covers why time to first token matters for interactive endpoints.
Where the money goes: billing phases compared

The diagram shows why two platforms with similar hourly rates can produce very different bills. Modal's pricing FAQ says it bills application load time, processing time and an idle window that defaults to 60 seconds; its cold start guide says scaledown_window can be set from 2 seconds to 20 minutes. Runpod's Serverless pricing docs bill cold start, execution and an idle timeout that defaults to 5 seconds, but say pulling the container image and downloading cached models normally happen while the worker is scaled down and are not billed.
Replicate's billing docs split by model type. Public models are billed only while processing, with setup and idle free. Most private models run on dedicated hardware and are billed for setup, idle and active time, with an exception for fast booting fine-tunes. Baseten's autoscaling docs bill each replica by the minute and keep idle replicas for a default scale-down delay of 900 seconds, configurable from 0 to 3,600. On Cloud Run, GPU services require instance-based billing, which charges for an instance's whole lifetime with a one-minute minimum, and Google's autoscaling docs say idle GPU instances can be kept for up to 10 minutes.
GPU prices compared
These are list prices for on-demand, scale-to-zero usage, converted to dollars per hour so they can be compared. Prices as of Nov 11, 2026, from each provider's official pricing page: Modal, Runpod, Replicate, Baseten and Google Cloud Run (us-central1).
Scroll the table horizontally on smaller screens.
| GPU | Modal (GPU only) | Runpod Serverless | Replicate | Baseten | Cloud Run (GPU only) |
|---|---|---|---|---|---|
| B200 (180 GB) | $6.25 | $10.79 | Not listed | $9.98 | Not offered |
| H200 (141 GB) | $4.54 | $6.99 | $5.49 (committed spend) | Not listed | Not offered |
| H100 (80 GB) | $3.95 | $5.39 | $5.49 | $6.50 | Not offered |
| RTX PRO 6000 (96 GB) | $3.03 | $3.49 | Not listed | Not listed | $1.31 (non-zonal) / $2.05 (zonal) |
| A100 (80 GB) | $2.50 | $2.72 | $5.04 | $4.00 | Not offered |
| L40S (48 GB) | $1.95 | $1.75 (48 GB tier) | $3.51 | Not listed | Not offered |
| L4 (24 GB) | $0.80 | $0.75 (24 GB tier) | Not listed | $0.85 | $0.67 (non-zonal) / $1.05 (zonal) |
| T4 (16 GB) | $0.59 | $0.58 (16 GB tier) | $0.81 | $0.63 | Not offered |
Read the table with three caveats:
- Modal and Cloud Run add CPU and memory. On Modal, a container with 4 physical cores and 32 GiB of memory adds about $0.44 an hour at list prices, taking an H100 to roughly $4.39. Cloud Run requires at least 4 vCPU and 16 GiB with an L4 (about $0.37 an hour extra) and at least 20 vCPU and 80 GiB with an RTX PRO 6000 (about $1.87 an hour extra), so an L4 instance costs about $1.05 an hour and an RTX PRO 6000 instance about $3.19 before zonal redundancy.
- Modal's list rates have multipliers. Its pricing page says Functions are preemptible by default, lists a 3x multiplier for non-preemptible execution and 1.15x to 1.75x for choosing a specific region.
- Runpod's flex rate is the standard rate. Its docs say active workers, which run 24/7 to avoid cold starts, are discounted through a sales inquiry, so always-on Runpod capacity can cost less than the list rate shown here.
Worked example: a bursty H100 endpoint
Here is what the billing rules do to a realistic workload. The numbers are illustrative, built from list prices and each platform's default settings, not from a test we ran.
The workload: one H100-class GPU serving a custom model for an internal agent tool. It is busy for 3 hours a day in 12 separate bursts, each burst starts from zero, and loading the model takes 60 seconds. Bursts are far enough apart that each one triggers a fresh cold start. Month = 30 days.
Scroll the table horizontally on smaller screens.
| Platform | Hourly rate used | Billed hours per day | Per day | Per month |
|---|---|---|---|---|
| Modal (4 cores, 32 GiB, 60 s idle) | $4.39 | 3.40 | $14.94 | $448 |
| Runpod Serverless (5 s idle) | $5.39 | 3.22 | $17.34 | $520 |
| Replicate private model (assumes 5 min idle) | $5.49 | 4.20 | $23.06 | $692 |
| Baseten (900 s default delay) | $6.50 | 6.20 | $40.30 | $1,209 |
| Baseten (delay lowered to 5 min) | $6.50 | 4.20 | $27.30 | $819 |
| Dedicated H100 SXM pod, 24/7 (Runpod Secure Cloud) | $3.99 | 24 | $95.76 | $2,873 |
Two lessons fall out of this. First, serverless beats an always-on GPU by a wide margin at low utilization, which is the whole point. Second, the idle window matters as much as the rate: Baseten's 15-minute default nearly doubles its billed hours here, and lowering it to 5 minutes cuts the bill by about a third at the price of more cold starts.
When dedicated GPUs win
Divide a dedicated GPU's daily cost by a serverless rate to get the break-even. Against a $3.99-an-hour H100 pod, Runpod Serverless breaks even at about 17.8 billed hours a day, Replicate at 17.4 and Baseten at 14.7. Modal's H100 rate is close to the pod price, so it stays competitive until roughly 21.8 billed hours a day, though its default preemptible execution is not the same guarantee as a dedicated machine. Above those levels, or when you need a warm GPU at all times, look at dedicated capacity. Our guide to spot vs reserved GPUs for inference covers how to buy it.
The platforms in detail
Modal
Modal runs Python functions in containers on its own multi-cloud GPU pool, with infrastructure declared in code. Its cold start guide says containers boot in about one second, and it offers min_containers and buffer_containers to keep warm capacity, plus memory snapshots to speed up initialization. The Starter plan is free with $30 of monthly compute and a cap of 10 concurrent GPUs; Team costs $250 a month with $100 of compute and 50 GPUs. Modal also sells per-token Model APIs for open-weight models.
Watch for: CPU and memory billed on top of the GPU, preemptible-by-default execution, and the 10-GPU concurrency cap on the free plan.
Runpod Serverless
Runpod Serverless runs your Docker container as queue-based workers with a handler function, or as load-balancing endpoints running your own HTTP server, according to Runpod's Serverless overview. It has the widest spread of GPU tiers, from $0.58 an hour for 16 GB cards to the B300, and the shortest default idle timeout. FlashBoot and cached models cut cold starts, and active workers keep a warm floor.
Watch for: a default account spend limit of $80 an hour across all resources, which support can raise, and tiered GPU pools where you may get any card in the tier.
Replicate
Replicate is strongest as a catalog: thousands of public models callable by API, many priced per output (for example per image) rather than per second. You can host your own models packaged with Cog, and deployments let you set minimum and maximum instances. Its hardware rates are higher than Modal's or Runpod's for the same GPU, but they include CPU and RAM.
Watch for: private models are billed for setup and idle time, which public models are not.
Baseten
Baseten combines per-minute dedicated deployments with per-token Model APIs. Its autoscaling docs default min_replica to 0, so deployments scale to zero, and expose the scale-down delay, concurrency target and maximum scale-down rate. Its listed GPU rates are the highest here, but the autoscaling controls suit production endpoints that need predictable behavior.
Watch for: the 900-second default scale-down delay, which you should tune for bursty traffic.
Google Cloud Run with GPUs
Cloud Run attaches one NVIDIA L4 (24 GB) or RTX PRO 6000 Blackwell (96 GB) per instance, and Google's GPU docs say instances with drivers pre-installed start in about 5 seconds. You get Cloud Run's usual developer experience, IAM and networking, and first deployments get a small automatic GPU quota (3 L4s per region with zonal redundancy off), with more by request. Zonal redundancy, on by default, costs more per second in exchange for better resilience during a zonal outage.
Watch for: one GPU per instance, no H100-class cards, a one-minute minimum per instance, and minimum CPU and memory that add to the GPU price.
Cloudflare Workers AI
Workers AI is not a GPU rental: you call models that Cloudflare hosts and pay per token, metered in Neurons. Its pricing page lists $0.011 per 1,000 Neurons above a free allocation of 10,000 Neurons a day. There are no cold starts or idle windows to manage, but you are limited to the catalog, which changes: our guide to the Workers AI model deprecations covers the latest retirements.
What about AWS?
AWS has no direct equivalent. SageMaker Serverless Inference does not support GPUs and tops out at 6 GB of memory. The closest option is a SageMaker real-time endpoint that hosts inference components, which can scale to zero instances; AWS warns that scaling back out takes several minutes, during which the endpoint can't serve those requests. For agents that run on AWS, our guide to deploying an AI agent on AWS covers the Lambda, ECS and Fargate side, and Amazon Bedrock covers per-token access to hosted models.
How to choose

Scroll the table horizontally on smaller screens.
| Situation | Start with | Why |
|---|---|---|
| A catalog model does the job | Workers AI or another per-token API | No GPUs, cold starts or idle billing |
| Custom model, spiky traffic, Python team | Modal | Lowest specialist H100 rate, code-defined infra |
| Custom container, cost-sensitive, mid-range GPU | Runpod Serverless | Low tier prices, 5 s default idle |
| Already on Google Cloud, model fits 24 or 96 GB | Cloud Run with GPUs | Same IAM, networking and deploy flow as the rest of your stack |
| Prototyping with many public models | Replicate | Setup and idle free on public models |
| Production endpoint that needs tuned autoscaling | Baseten | Replica-level autoscaling controls and model APIs |
| GPU busy most of the day | Dedicated or reserved GPUs | Cheaper above roughly 15 to 18 billed hours a day |
How to cut cold starts and idle cost
- Measure billed time, not busy time. Track cold start duration and idle seconds per request burst; that ratio tells you whether to tune the idle window or move to dedicated capacity.
- Keep weights close to the GPU. Use the platform's model cache or volumes rather than downloading weights from the internet on every cold start.
- Tune the idle window to your traffic gaps. If bursts arrive every 2 minutes, a 60-second window pays a cold start every time; a 3-minute window may be cheaper overall.
- Keep one warm container only where latency matters. A single minimum instance on an H100 costs thousands of dollars a month at list prices, so reserve it for user-facing endpoints.
- Batch where you can. Queue-based endpoints and batch jobs pack more work into each billed second than one-request-per-container serving.
- Using TensorRT-LLM? There is no engine build anymore. NVIDIA's migration guide (last updated Nov 3, 2026) says the TensorRT engine backend has been removed: Hugging Face checkpoints load directly with trtllm-serve, with no trtllm-build or checkpoint-conversion step to bake into your container image. Our vLLM vs SGLang vs TensorRT-LLM comparison covers the engines in more depth.
Here is an illustrative Modal configuration that keeps a GPU endpoint at zero when idle and holds containers for 3 minutes between bursts. It was not run for this article; check Modal's docs for current parameter names.
# Illustrative only: a scale-to-zero GPU endpoint on Modalimport modalapp = modal.App("inference-demo")image = modal.Image.debian_slim().pip_install("vllm")@app.function(gpu="H100", image=image, min_containers=0, # scale to zero when idle scaledown_window=180) # keep containers 3 min after the last request@modal.fastapi_endpoint(method="POST")def generate(item: dict): ... # load the model once per container, then serve requests
Verdict
Serverless GPUs are the right default for custom models with bursty traffic, internal agent tools, demos and batch jobs. Pick on billing rules as much as on rate. Modal offers the best H100 price among the specialists and a clean Python workflow; Runpod Serverless is cheapest for mid-range GPUs and wastes the least on idle time by default; Cloud Run with GPUs makes sense if your stack already lives on Google Cloud and your model fits an L4 or RTX PRO 6000.
Use Replicate when you mainly call public models, Baseten when you want managed autoscaling for production, and a per-token API like Workers AI whenever a catalog model is good enough. Track billed GPU-hours from day one: once they pass roughly 15 to 18 a day, it is time to price dedicated capacity.
Sources
- Modal: Pricing (accessed Nov 11, 2026)
- Modal: Cold start performance (accessed Nov 11, 2026)
- Runpod: Pricing (accessed Nov 11, 2026)
- Runpod docs: Serverless pricing (last modified Oct 15, 2026)
- Runpod docs: Serverless overview (accessed Nov 11, 2026)
- Replicate: Pricing (accessed Nov 11, 2026)
- Replicate docs: How billing works (accessed Nov 11, 2026)
- Baseten: Pricing (accessed Nov 11, 2026)
- Baseten docs: Autoscaling (accessed Nov 11, 2026)
- Google Cloud: Cloud Run pricing (accessed Nov 11, 2026)
- Google Cloud: Configure GPUs for Cloud Run services (accessed Nov 11, 2026)
- Google Cloud: About instance autoscaling in Cloud Run (accessed Nov 11, 2026)
- Cloudflare: Workers AI pricing (accessed Nov 11, 2026)
- AWS: SageMaker Serverless Inference (accessed Nov 11, 2026)
- AWS: Scale a SageMaker endpoint to zero instances (accessed Nov 11, 2026)
Tools covered in this guide
Frequently asked questions
- What is serverless GPU inference?
- It is a way to serve a machine learning model where the platform starts GPU containers when requests arrive and shuts them down when traffic stops, so you pay per second or per minute of GPU time instead of renting a GPU around the clock. Modal, Runpod Serverless, Replicate, Baseten and Cloud Run with GPUs all work this way.
- Which serverless GPU provider is cheapest?
- It depends on the GPU and your traffic. As of November 11, 2026, Modal lists the lowest H100 rate among the specialists at about $3.95 an hour plus CPU and memory, Runpod Serverless is cheapest on many mid-range tiers, and Cloud Run's L4 is $0.67 an hour for the GPU alone. Idle windows and cold starts can change the ranking for bursty workloads.
- Do you pay for cold starts on serverless GPUs?
- Usually yes. Modal, Runpod Serverless, Baseten, Cloud Run and private Replicate models all bill the time a container spends starting and loading the model. Replicate's public models are the exception: setup and idle time are free. Runpod says image pulls and cached model downloads normally happen while workers are scaled down and are not billed.
- Does AWS have serverless GPU inference?
- Not directly. SageMaker Serverless Inference does not support GPUs. SageMaker real-time endpoints that host inference components can scale to zero instances, but AWS says scaling back out takes several minutes. Amazon Bedrock offers per-token access to hosted models instead.
- When should I switch from serverless GPUs to a dedicated GPU?
- When your billed GPU time gets close to the break-even point. Against a $3.99-an-hour dedicated H100, serverless list rates break even at roughly 15 to 18 billed hours a day, or about 22 hours for Modal. A hard latency requirement that needs a warm GPU at all times also favors dedicated or reserved capacity.