Cold starts

A cold start is the time a fresh replica spends starting up before it can accept traffic. By default, a synchronous request that triggers a cold start waits until the replica is ready, so startup time becomes part of the request’s latency. The following diagram traces a deployment through that cycle, from Scaled to zero to Active and back, with the startup steps that add up to the wait.

Cold start triggers

Every new replica cold-starts before it can serve traffic, no matter why it was created. Scale-from-zero applies when a deployment’s min_replica is 0. After traffic drops, the autoscaler waits for scale_down_delay, removes replicas up to max_scale_down_rate, and resets the delay. It repeats these steps until no replicas remain. A synchronous request can then trigger a replica startup and wait for it to finish. Scaling events happen while a deployment is already serving traffic. When load crosses the scaling threshold, the autoscaler adds replicas, and each one cold-starts before it can serve traffic. The replicas already running keep serving in the meantime, so users notice only when load grows faster than new replicas can start up.

Contributing factors

A new replica works through these steps in order, and their durations add up to the cold-start time:

Step	What happens
Container pull	The replica downloads your Docker image layers.
Weight load	Model weights move from storage into accelerator memory.
Engine initialization	Your model’s setup code runs. For inference engines like vLLM and SGLang, this includes capturing CUDA graphs, compiling kernels with `torch.compile`, and profiling the KV cache.

Baseten provides the Baseten Delivery Network (BDN), which speeds up weight load by mirroring your weights and caching them near your replicas. Each scale-up can read the weights from that cache instead of downloading them again from the source. Baseten also streams your container image in the background so image transfer can overlap other startup work. The dominant step depends on the model, runtime, and hardware. Engine initialization can dominate when graph capture or compilation is substantial. Weight loading can dominate when the model has larger weight files. Benchmark your deployment to identify the step to optimize.

Reduce cold starts

The biggest win comes from shrinking whichever step dominates startup. When that isn’t enough, keep replicas warm so requests skip the cold start entirely.

Faster weight loading

BDN runs automatically on engine-builder deployments. On any other deployment, turn it on by adding a weights block to your config.

Compilation caching

torch.compile creates artifacts while a replica starts. Torch compile caching, built on b10cache, persists those artifacts so a new replica can reuse them instead of compiling from scratch. Benchmark your deployment to measure the effect on startup time.

Warm replicas

min_replica sets a floor on running replicas. Keep it at 1 or higher so a replica stays warm to serve the first request. You pay for that replica while it’s idle, but requests avoid a scale-from-zero wait or rejection. Set it in the dashboard or through the autoscaling settings API:

Autoscaling settings

{
  "min_replica": 1
}

For production redundancy, set min_replica to 2 or higher so another warm replica remains available while Baseten replaces a failed or restarting replica. Your replica floor trades cost against latency:

Approach	Cost	Latency	Best for
Scale to zero (`min_replica: 0`)	No running-replica charges while idle; wake-up minutes are billed	Synchronous requests can wait for a replica to start.	Batch jobs, development, and spiky low-volume traffic
Always on (`min_replica` ≥ 1)	Pay for idle replicas	No cold start from idle, though new replicas still cold-start	Latency-sensitive production traffic

Start warm for production, and scale to zero only when an occasional cold-start wait or retry is acceptable.

Pre-warming

For predictable traffic spikes, raise min_replica ahead of the expected load:

Terminal

# Run before the spike. Use your observed cold start as the lead time.
curl -X PATCH \
  https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/autoscaling_settings \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{"min_replica": 5}'

After traffic stabilizes, reset to your normal minimum.

Scale-down delay

A longer scale-down delay keeps replicas warm through brief traffic dips. The default is 15 minutes (900 seconds); this example doubles it to 30 minutes:

Autoscaling settings

{
  "scale_down_delay": 1800
}

A replica that’s still warm when traffic returns serves immediately, with no cold start.

Next steps

Request lifecycle: What happens to requests during cold starts, including queuing and timeout behavior.
Autoscaling: Configure min_replica, scale_down_delay, and the rest of the scaling settings.
Traffic patterns: Pre-warming strategies for different traffic types.
Billing and usage: How cold-start time is metered.
Troubleshooting: Diagnose cold start issues.

Get started

Model APIs

Inference

Training

Development

Deployment

Engines

Model Labs

Organization

Hosting options

Observability

Troubleshooting

Cold start triggers

Contributing factors

Reduce cold starts

Faster weight loading

Compilation caching

Warm replicas

Pre-warming

Scale-down delay

Next steps

​Cold start triggers

​Contributing factors

​Reduce cold starts

​Faster weight loading

​Compilation caching

​Warm replicas

​Pre-warming

​Scale-down delay

​Next steps

Cold start triggers

Contributing factors

Reduce cold starts

Faster weight loading

Compilation caching

Warm replicas

Pre-warming

Scale-down delay

Next steps