Scale a deployment

Scale your deployments as traffic hits your endpoint. Traffic can change faster than you can react to it: pre-scale ahead of the spikes you can see coming, and let the autoscaler absorb the ones you can’t. This page covers applying scaling changes to a live deployment. To choose the values themselves, start with the autoscaling overview and match the settings to your traffic patterns.

Update autoscaling settings

Update autoscaling settings in place to pre-scale for a known traffic spike, raise a replica ceiling, or tune scale-down behavior. The change applies to the running deployment; replicas adjust without a new deploy.

UI
REST API

To update autoscaling settings:

Sign in to your workspace at app.baseten.co and choose Dedicated Inference in the sidebar.
Select your model.
On the model’s overview, choose Configure under the environment’s Replicas.
Adjust your settings:
- Replicas: the minimum and maximum replica count. Baseten scales within this range based on traffic.
- Autoscaling window: how far back the autoscaler looks when averaging traffic for scaling decisions.
- Scale down delay: how long the autoscaler waits after traffic drops before removing replicas.
- Max scale down rate: the largest percentage of active replicas the autoscaler removes in a single scale-down step.
- Concurrency target: the number of concurrent requests each replica handles before the autoscaler adds another.
- Target utilization percentage: the share of the concurrency target at which scaling triggers.
Choose Update to apply the changes.

The same dialog is in the environment card’s ⋯ menu as Configure autoscaling.

To update autoscaling settings:Use the autoscaling_settings endpoint to apply autoscaling updates. It accepts any subset of the fields, so send only the ones you’re changing; the update applies asynchronously:

curl -X PATCH "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/autoscaling_settings" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "min_replica": 2,
    "max_replica": 8,
    "autoscaling_window": 60,
    "scale_down_delay": 900,
    "concurrency_target": 2,
    "target_utilization_percentage": 70
  }'

{
  "status": "ACCEPTED",
  "message": "Your request to update autoscaling settings has been accepted. Query for deployment {deployment_id}'s status to see when the updates have been applied."
}

Now that your deployment’s settings are updated, verify the changes reached the deployment. The update applies asynchronously, so the new values can take a moment to land:

UI
Baseten CLI
REST API

To verify your settings:

Sign in to your workspace at app.baseten.co and choose Dedicated Inference in the sidebar.
Select your model.
Select the deployment under Deployments.
Review the current values on the Autoscaling settings card: replicas, autoscaling window, scale down delay, concurrency target, and target utilization.

To verify your settings:

baseten model deployment describe --model-id <model-id> --deployment-id <deployment-id> --jq '.autoscaling_settings'

{
  "autoscaling_window": 60,
  "concurrency_target": 2,
  "max_replica": 8,
  "min_replica": 2,
  "scale_down_delay": 900,
  "target_utilization_percentage": 70
}

To verify your settings:

curl "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}" \
  -H "Authorization: Bearer $BASETEN_API_KEY"

{
  "id": "def456",
  "autoscaling_settings": {
    "min_replica": 2,
    "max_replica": 8,
    "autoscaling_window": 60,
    "scale_down_delay": 900,
    "concurrency_target": 2,
    "target_utilization_percentage": 70
  },
  ...
}

For more information, see the update autoscaling settings endpoint. To update settings on whatever deployment an environment currently serves, use the environment settings endpoint instead.

Scale back down

After a spike passes, lower min_replica back to its normal floor with the same update procedure. Two settings control how fast replicas drain:

Scale down delay (scale_down_delay): how long the autoscaler waits after traffic drops before removing replicas.
Max scale down rate: the largest percentage of active replicas removed in a single step. Set it in the console’s Configure autoscaling dialog.

For example, drop the floor back to one replica and keep a 15-minute drain delay:

curl -X PATCH "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/autoscaling_settings" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"min_replica": 1, "scale_down_delay": 900}'

{
  "status": "ACCEPTED",
  "message": "Your request to update autoscaling settings has been accepted. Query for deployment {deployment_id}'s status to see when the updates have been applied."
}

Replicas above the new floor drain gradually; traffic keeps flowing to the replicas that remain.

Scale to zero

Set min_replica to 0 to let an idle deployment release all its replicas and stop billing for compute.

Scaling to zero isn’t recommended for production endpoints: the first request after an idle period pays a cold start. Reserve it for development and staging deployments, or for workloads that tolerate the delay.

UI
REST API

To scale to zero:

Sign in to your workspace at app.baseten.co and choose Dedicated Inference in the sidebar.
Select your model.
On the model’s overview, choose Configure under the environment’s Replicas.
Set Min replicas to 0.
Choose Update to apply the changes.

To scale to zero:

curl -X PATCH "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/autoscaling_settings" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"min_replica": 0}'

{
  "status": "ACCEPTED",
  "message": "Your request to update autoscaling settings has been accepted. Query for deployment {deployment_id}'s status to see when the updates have been applied."
}

The model’s endpoint stays live, and the next request spins a replica back up. To stop serving entirely, deactivate the deployment instead.

Wake a scaled-to-zero deployment

Wake a scaled-to-zero deployment before you need it, for example ahead of a demo or a batch job, so the first real request doesn’t pay the cold start.

UI
REST API

To wake a deployment:

Sign in to your workspace at app.baseten.co and choose Dedicated Inference in the sidebar.
Select your model.
Select the deployment under Deployments.
Choose Wake deployment.

The environment card has the same action as Wake.

To wake a deployment:

curl -i -X POST "https://model-{model_id}.api.baseten.co/deployment/{deployment_id}/wake" \
  -H "Authorization: Bearer $BASETEN_API_KEY"

HTTP/2 202
content-length: 0

The deployment starts a replica in the background, moving from SCALED_TO_ZERO through WAKING_UP to ACTIVE; poll baseten model deployment describe until status is ACTIVE. A woken deployment with min_replica: 0 scales back down after scale_down_delay if no requests arrive, so wake it close to when you need it, or raise min_replica to hold it warm.

Next steps

Scaling changes take effect on the running deployment, so pair them with a quick status check before and after.

Autoscaling overview to choose replica counts, concurrency targets, and scale-down behavior.
Manage the deployment lifecycle to deactivate, promote, or delete deployments.
Pull logs and metrics to confirm how a scaling change lands.

​Update autoscaling settings

​Scale back down

​Scale to zero

​Wake a scaled-to-zero deployment

​Next steps

Update autoscaling settings

Scale back down

Scale to zero

Wake a scaled-to-zero deployment

Next steps