Scale a deployment

Use this page to update autoscaling settings on a live deployment, scale it to zero, or wake it. To choose the values, start with the autoscaling overview and match the settings to your traffic patterns.

Update autoscaling settings

Update autoscaling settings in place to pre-scale for a known traffic spike, raise a replica ceiling, or tune scale-down behavior. The change applies to the running deployment; replicas adjust without a new deploy.

UI
Baseten CLI
Management API

To update autoscaling settings:

Sign in to your workspace at app.baseten.co and choose Dedicated Inference in the sidebar.
Select your model.
On the model’s overview, choose Configure under the environment’s Replicas.
Adjust your settings:
- Replicas: the minimum and maximum replica count. Baseten scales within this range based on traffic.
- Autoscaling window: how far back the autoscaler looks when averaging traffic for scaling decisions.
- Scale down delay: how long the autoscaler waits after traffic drops before removing replicas.
- Max scale down rate: the largest percentage of active replicas the autoscaler removes in a single scale-down step.
- Concurrency target: the number of concurrent requests each replica handles before the autoscaler adds another.
- Target utilization percentage: the share of the concurrency target at which scaling triggers.
Choose Update to apply the changes.

The same dialog is in the environment card’s ⋯ menu as Configure autoscaling.

To update autoscaling settings:Pass only the flags you want to change; every other setting is left alone. The update applies asynchronously:

baseten model deployment update-autoscaling \
  --model-id <model-id> --deployment-id <deployment-id> \
  --min-replica 2 --max-replica 8

ACCEPTED
Your request to update autoscaling settings has been accepted. Query for deployment <deployment-id>'s status to see when the updates have been applied.

See baseten model deployment update-autoscaling for every flag.

To update autoscaling settings:Use the autoscaling_settings endpoint to apply autoscaling updates. It accepts any subset of the fields, so send only the ones you’re changing; the update applies asynchronously:

curl -X PATCH "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/autoscaling_settings" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "min_replica": 2,
    "max_replica": 8,
    "autoscaling_window": 60,
    "scale_down_delay": 900,
    "concurrency_target": 2,
    "target_utilization_percentage": 70
  }'

{
  "status": "ACCEPTED",
  "message": "Your request to update autoscaling settings has been accepted. Query for deployment {deployment_id}'s status to see when the updates have been applied."
}

Autoscaling updates apply asynchronously. Verify the deployment after you submit a change:

UI
Baseten CLI
Management API

To verify your settings:

Sign in to your workspace at app.baseten.co and choose Dedicated Inference in the sidebar.
Select your model.
Select the deployment under Deployments.
Review the current values on the Autoscaling settings card: replicas, autoscaling window, scale down delay, concurrency target, and target utilization.

To verify your settings:

baseten model deployment describe --model-id <model-id> --deployment-id <deployment-id> --jq '.autoscaling_settings'

{
  "autoscaling_window": 60,
  "concurrency_target": 2,
  "max_replica": 8,
  "min_replica": 2,
  "scale_down_delay": 900,
  "target_utilization_percentage": 70
}

To verify your settings:

curl "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}" \
  -H "Authorization: Bearer $BASETEN_API_KEY"

{
  "id": "def456",
  "autoscaling_settings": {
    "min_replica": 2,
    "max_replica": 8,
    "autoscaling_window": 60,
    "scale_down_delay": 900,
    "concurrency_target": 2,
    "target_utilization_percentage": 70
  },
  ...
}

For more information, see the update autoscaling settings endpoint. To update settings on whatever deployment an environment currently serves, use the environment settings endpoint instead.

Scale back down

After a spike passes, lower min_replica back to its normal floor with the same update procedure. Two settings control how fast replicas drain:

Scale down delay (scale_down_delay): how long the autoscaler waits after traffic drops before removing replicas.
Max scale down rate: the largest percentage of active replicas removed in a single step. Set it in the dashboard’s Configure autoscaling dialog.

For example, drop the floor back to one replica and keep a 15-minute drain delay:

curl -X PATCH "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/autoscaling_settings" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"min_replica": 1, "scale_down_delay": 900}'

{
  "status": "ACCEPTED",
  "message": "Your request to update autoscaling settings has been accepted. Query for deployment {deployment_id}'s status to see when the updates have been applied."
}

Replicas above the new floor drain gradually; traffic keeps flowing to the replicas that remain.

Scale to zero

Set min_replica to 0 to let an idle deployment release all its replicas and stop billing for compute.

Scaling to zero adds a cold start when traffic returns. Synchronous requests can wait while a replica starts. Use scale-to-zero only when your workload can tolerate that wait.

UI
Management API

To scale to zero:

Sign in to your workspace at app.baseten.co and choose Dedicated Inference in the sidebar.
Select your model.
On the model’s overview, choose Configure under the environment’s Replicas.
Set Min replicas to 0.
Choose Update to apply the changes.

To scale to zero:

curl -X PATCH "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/autoscaling_settings" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"min_replica": 0}'

{
  "status": "ACCEPTED",
  "message": "Your request to update autoscaling settings has been accepted. Query for deployment {deployment_id}'s status to see when the updates have been applied."
}

The model’s endpoint stays live while the deployment has no replicas. To stop serving entirely, deactivate the deployment instead.

Wake a scaled-to-zero deployment

Wake a scaled-to-zero deployment before you need it, for example ahead of a demo or a batch job, so traffic avoids a cold-start wait or capacity rejection.

UI
Inference API

To wake a deployment:

Sign in to your workspace at app.baseten.co and choose Dedicated Inference in the sidebar.
Select your model.
Select the deployment under Deployments.
Choose Wake deployment.

The environment card has the same action as Wake.

To wake a deployment:

curl -i -X POST "https://model-{model_id}.api.baseten.co/deployment/{deployment_id}/wake" \
  -H "Authorization: Bearer $BASETEN_API_KEY"

HTTP/2 202
content-length: 0

The deployment starts a replica in the background, moving from SCALED_TO_ZERO through WAKING_UP to ACTIVE; poll baseten model deployment describe until status is ACTIVE. A woken deployment with min_replica: 0 scales back down after scale_down_delay if no requests arrive, so wake it close to when you need it, or raise min_replica to hold it warm.

Next steps

Scaling changes take effect on the running deployment, so pair them with a quick status check before and after.

Autoscaling overview to choose replica counts, concurrency targets, and scale-down behavior.
Manage the deployment lifecycle to deactivate, promote, or delete deployments.
Pull logs and metrics to confirm how a scaling change lands.

Get started

Model APIs

Inference

Training

Development

Deployment

Engines

Model Labs

Organization

Hosting options

Observability

Troubleshooting

Scale a deployment

Update autoscaling settings

Scale back down

Scale to zero

Wake a scaled-to-zero deployment

Next steps

​Update autoscaling settings

​Scale back down

​Scale to zero

​Wake a scaled-to-zero deployment

​Next steps

Update autoscaling settings

Scale back down

Scale to zero

Wake a scaled-to-zero deployment

Next steps