Deployments

​

Issue: truss push can’t find config.yaml

[Errno 2] No such file or directory: '/path/to/your-truss/config.yaml'

​

Fix: set correct target directory

• Running truss push from the directory containing the Truss. You should see the file config.yaml when you run ls in your working directory.
• Or passing the target directory as an argument, such as truss push /path/to/my-truss.

​

Issue: unexpected failure during model build

​

Fix: restart deploy from Baseten UI

​

Issue: a single replica is stuck or unhealthy

​

Fix: terminate the affected replica

baseten model deployment logs --model-id <model-id> --deployment-id <deployment-id> --since 1h --jq '.replica'

""
"nvsmp"

baseten model deployment replica terminate --model-id <model-id> --deployment-id <deployment-id> --replica-id <replica-id>

Terminated replica <replica-id> of deployment <deployment-id>

curl "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/logs" \
  -H "Authorization: Bearer $BASETEN_API_KEY"

{
  "logs": [
    {
      "timestamp": "1783445624380306972",
      "message": "Completed model.load() execution in 6 ms",
      "replica": "nvsmp",
      "request_id": null,
      "level": "INFO"
    }
  ]
}

curl -X DELETE "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/replicas/{replica_id}" \
  -H "Authorization: Bearer $BASETEN_API_KEY"

{"success": true}

​

Issue: requests fail with “Model version is deactivated”

Model version {deployment_id} is deactivated. It needs to be activated before running predictions

​

Fix: activate the deployment

baseten model deployment activate --model-id <model-id> --deployment-id <deployment-id>

​

Issue: “You’ve reached the maximum number of deployed models”

You've reached the maximum number of deployed models (2000) for your tier. Either remove older model deployments or email us at support@baseten.co. We are happy to give you a higher limit.

• It counts deployments, not models: A model with four deployments uses four slots. Chainlet deployments count individually, so a chain with five chainlets uses five slots.
• Scaled-to-zero deployments count: A deployment scaled to zero runs no replicas and costs nothing, but its endpoint stays live and ready to cold-start, so it holds its slot. Only deactivated deployments release theirs.

• ACTIVE
• SCALED_TO_ZERO
• WAKING_UP
• UPDATING
• UNHEALTHY
• BUILDING
• DEPLOYING

• INACTIVE
• DEACTIVATING
• BUILD_FAILED
• DEPLOY_FAILED
• BUILD_STOPPED

​

Fix: free deployment slots

for id in $(baseten model list --output json | jq -r '.models[].id'); do
  baseten model deployment list --model-id "$id" --output json \
    | jq -r --arg m "$id" '.deployments[]
        | select(.status | IN("ACTIVE","SCALED_TO_ZERO","WAKING_UP","UPDATING","UNHEALTHY","BUILDING","DEPLOYING"))
        | "\($m) \(.id) \(.status) \(.active_replica_count)"'
done

abc123 def456 SCALED_TO_ZERO 0
abc123 ghi789 ACTIVE 1
jkl012 mno345 SCALED_TO_ZERO 0

1. Deactivate deployments you don’t expect to serve traffic soon: Deactivating keeps the deployment’s configuration and releases its slot, and you can activate it again later. This is the fastest way to unblock a push.
2. Delete deployments you’re finished with: Deleting is irreversible, so reserve it for finished experiments and stale versions.
3. Stop accumulating superseded deployments: After a promotion, the target environment’s promotion_cleanup_strategy decides what happens to the deployment it replaced. The default, SCALE_TO_ZERO, leaves that deployment holding its slot. Set the strategy to DEACTIVATE so each promotion releases the slot it supersedes. See deployment cleanup.
4. Request a higher limit: If every counted deployment is one you need, email support@baseten.co.

​

Autoscaling issues

​

Latency spikes during scaling events

• Replicas terminated while handling in-flight requests
• Cold start delays while new replicas initialize

1. Increase scale-down delay (for example, 300s → 900s) to reduce how often replicas are removed.
2. Increase min replicas to reduce cold start frequency.
3. Lower target utilization to provide more headroom during scaling.

​

Replicas oscillating (thrash)

1. Increase scale-down delay: this is the primary lever for oscillation.
2. Increase autoscaling window to smooth out noise.
3. Lower max scale-down rate so each scale-down step removes fewer replicas.
4. Only then consider lowering target utilization for more headroom.

​

Slow scale-up / “Scaling up replicas” persists

• GPU capacity not available in your region
• Slow model initialization (large weights, slow downloads)

1. Pre-warm by bumping min replicas through the API before expected load spikes.
2. Contact support about capacity pool availability.
3. Check if optimized images are being used (look for “streaming-enabled image” in logs).

​

Model scales to zero before testing

​

Async queue growing without bound

1. Increase max replicas to add more processing capacity.
2. Increase concurrency target if your model can handle more concurrent requests.
3. Lower target utilization to trigger scaling earlier.

​

Bill higher than expected

1. Raise concurrency target to squeeze more throughput from each replica.
2. Monitor p95 latency as you raise concurrency. If latency stays stable, keep raising; if it rises sharply, you’ve gone too far.
3. Enable scale-to-zero (min replicas = 0) for intermittent workloads.
4. Review your traffic patterns and adjust settings accordingly. See Traffic patterns.

​

Cold starts taking too long

• Large model weights (10s–100s of GB)
• Slow network downloads from model registries
• Heavy initialization code in load() method

1. Look for “streaming-enabled image” in logs. This confirms image streaming is active.
2. Keep min_replica ≥ 1 to avoid cold starts entirely.
3. Pre-warm before expected traffic spikes using the autoscaling API.

​

Development deployment won’t scale

​

Not sure which traffic pattern I have

1. Sign in to your workspace at app.baseten.co and choose Dedicated Inference in the sidebar, then select your model.
2. Choose the Metrics tab.
3. Look at Inference volume and Replicas over the past week.
4. Identify your pattern:

​

Concurrency target misconfigured

• Too low (common): Running many more replicas than needed. Default of 1 is conservative but expensive.
• Too high: Requests queue at replicas, causing latency even when replica count looks healthy.

1. Benchmark your model to find actual throughput capacity.
2. Use starting points by model type:

3. Gradually increase while monitoring p95 latency. Stop when latency rises sharply.