Deploy and iterate

Use development deployments to test changes to a Python Truss model. The Baseten CLI watches your project and applies supported edits to the running deployment without a full rebuild. Custom Docker servers don’t support development deployments. For a Docker server or engine configuration, use Publish and validate an update, then promote the validated deployment.

SSH access lets you connect to a running deployment with standard SSH. Pair it with a non-zero min_replicas and a code-syncing tool when you need an interactive session inside a custom server.

Prepare a Python Truss project

You’ll need a Baseten account, the Baseten CLI, and a CLI profile authenticated with baseten auth login. Use an account with permission to deploy models. Keep one terminal open for the watcher and use a second terminal for prediction requests. Install jq to extract the published deployment ID during release. This example creates billable CPU compute. Deactivate the deployments when you’re done. Create the project directory, then save the following files:

Terminal

mkdir -p hello-world/model
cd hello-world

config.yaml

model_name: HelloWorld
python_version: py311
requirements: []
resources:
  cpu: "1"
  memory: 2Gi
  use_gpu: false

model/model.py

class Model:
    def predict(self, model_input):
        return {"greeting": f"Hello, {model_input['name']}!"}

Choose an unused model_name if HelloWorld already exists in your team. The predict() method accepts a JSON object with a name field and returns a greeting. You can use your own Python Truss project instead; use an input that matches its prediction method in the requests below.

Start a development deployment

Run the following command from the project directory to create a development deployment and watch for changes:

baseten model push --watch

Pushing model "HelloWorld"...
Uploading model...
Uploaded model in 0s
✨ Model HelloWorld was successfully pushed ✨

  Model:      HelloWorld (abc1d2ef)
  Deployment: xyz123

🪵 View logs:
   deployment:  baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   app:         https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/development/predict
   CLI:  baseten model predict --model-id abc1d2ef
Waiting for deployment to be ready...
Status: BUILDING
Status: DEPLOYING
Status: LOADING_MODEL
Watching "HelloWorld" (development deployment xyz123)...
Watching for changes. Press Ctrl-C to stop.

Copy your model and deployment IDs from the output. The example uses model ID abc1d2ef and development deployment ID xyz123. The summary’s CLI hint omits the request body and environment; use the complete prediction command below to call development. If deployment fails, inspect the build or deployment logs and follow deployment troubleshooting. A build failure occurs before the container starts; a startup failure occurs while the model loads.

Test an edit

To verify a code change:

In your second terminal, send a request to the development environment. Replace abc1d2ef with your model ID:

baseten model predict --model-id abc1d2ef --environment development --data '{"name": "Baseten"}'

{
  "greeting": "Hello, Baseten!"
}

Replace model/model.py with the following code and save it:

model/model.py

class Model:
    def predict(self, model_input):
        return {"greeting": f"Welcome, {model_input['name']}!"}

Wait for the watcher to finish applying the patch, then send the same request again:

baseten model predict --model-id abc1d2ef --environment development --data '{"name": "Baseten"}'

{
  "greeting": "Welcome, Baseten!"
}

The changed greeting confirms that the development deployment is running your edit. To apply model code changes without restarting the inference server, stop the watcher and run the following command:

Terminal

baseten model push --watch --watch-hot-reload

For more information, see What gets live-patched.

Re-attach to a development deployment

Ctrl+C stops the local watcher but leaves the development deployment running. To sync changes and resume watching it, run the following command from the project directory:

Terminal

baseten model watch

The watcher reports the existing deployment:

Watching "HelloWorld" (development deployment xyz123)...
Waiting for deployment to be ready...
Status: DEPLOYING
Status: LOADING_MODEL
💤 Keeping development deployment warm
Development deployment already up to date.
Watching for changes. Press Ctrl-C to stop.

baseten model watch syncs any changes made while disconnected, then resumes watching. It requires an existing development deployment. If you don’t have one, use baseten model push --watch to create it. To resume watching with hot reload, run the following command:

Terminal

baseten model watch --hot-reload

What gets live-patched

The Baseten CLI monitors your project directory and applies patches for the following changes without a full rebuild:

Change type	Examples
Model code	Files in the `model/` directory: `model.py`, helper modules, utilities, and binary files (like `.so`, `.png`).
Bundled packages	Files in the `packages/` directory, including binary files (like `.pyd`, `.so`).
Python requirements	Adding, removing, or updating packages in `requirements` or a requirements file.
Environment variables	Adding, removing, or updating values in `environment_variables`.
External data	Adding or removing entries in `external_data`.
Config values	Most `config.yaml` changes (except those listed below).

With the --watch-hot-reload or --hot-reload flags, the Baseten CLI hot-reloads model code changes by swapping the model class in-process without restarting the inference server. This preserves in-memory state like loaded weights and caches. If a patch includes non-model changes (such as requirements or config), it falls back to a standard restart.

Hot reload re-imports your module and updates __class__ on the existing model instance. It doesn’t re-run __init__() or load(). If you add new instance state in those methods that predict() depends on, predict() calls can’t see it. When your changes involve new instance state, stop the watch session and do a full reload with baseten model push --watch.

What requires a full redeploy

The patch system doesn’t support some changes. When you make these changes, stop the watch session and run baseten model push (or baseten model push --watch to start a new development deployment):

Change type	Why
`resources` (GPU type, count)	Requires a new instance.
`python_version`	Requires a new base image.
`system_packages`	Requires apt installation in the container.
`live_reload`	Changes the deployment mode.
Data directory (`data/`)	The patch system doesn’t track file changes in `data/`.

If a patch fails, the watcher prints an error and continues watching. Fix the issue in your source files and save again. For persistent failures, run baseten model push --watch to start fresh.

Limitations

Development deployments optimize for iteration, not production traffic:

Single replica: Fixed at 0 minimum, 1 maximum. No autoscaling beyond one replica.
No gRPC: Trusses with gRPC transport require a published deployment.
No custom Docker servers or TRT-LLM engine builds: Use a published deployment for these configurations.

See Development deployments for the full autoscaling constraints.

Publish and validate an update

Publish an immutable deployment when you’re ready to release a Python Truss model or update a custom Docker server or engine configuration. Use the same project and model_name as the model you want to update. If the model already has a production deployment, a push without an environment creates a candidate; promote it explicitly to update production. When the model has no production deployment, Baseten assigns its first published deployment to production. For the greeting model in this walkthrough, the first publish therefore creates the initial production release. The request below validates that deployment directly. To publish and test a candidate:

Save a valid input for your model as request.json. For the Python greeting model, use:

request.json

{"name": "Baseten"}

For the vLLM Qwen server, use:

request.json

{
  "model": "Qwen/Qwen2.5-3B-Instruct",
  "messages": [
    {"role": "user", "content": "Explain inference in one sentence."}
  ],
  "max_tokens": 128
}

Stop the watcher if it’s running. From the project directory, publish the current files and save the new deployment ID:
Terminal
published_deployment_id="$(baseten model push --tail --wait --output json | jq -r '.deployment.id')"
The command streams build and startup logs, waits for the deployment to become Active, and saves its ID. If the push fails, inspect the logs before continuing.
Send your test input to the published deployment. Replace abc1d2ef with your model ID:
Terminal
baseten model predict --model-id abc1d2ef --deployment-id "$published_deployment_id" --file request.json
The greeting model returns {"greeting": "Welcome, Baseten!"}. For your own model, confirm the response matches the behavior you intend to release.

The deployment ID targets the published files you just pushed, regardless of which deployment currently serves production.

Deploy to production

For an update to an existing production release, promote the validated deployment using the ID in published_deployment_id. Save the ID returned by promotion and follow the same procedure’s environment check. Promotion can return a different ID, and it can finish after the command returns. For the first published release, skip promotion and use published_deployment_id in the environment check: Baseten has already selected that deployment for production. After the production environment’s current_deployment.id matches the promoted ID, send the same input through the stable endpoint:

baseten model predict --model-id abc1d2ef --environment production --file request.json

{
  "greeting": "Welcome, Baseten!"
}

The matching response confirms that production serves your edit. Published deployments support autoscaling beyond the development deployment’s single replica.

Clean up

When you’ve finished the example, deactivate each deployment you created, including the development deployment and any published or promoted deployments. Confirm each deployment reaches Inactive. Stopping the watcher doesn’t stop the deployment or its compute spend.

CLI reference: model push

Full list of options for the push command.

CLI reference: model watch

Full list of options for the watch command.

Autoscaling

Configure replicas, concurrency targets, and scale-to-zero for production.

Environments

Manage staging, production, and custom environments.

Get started

Model APIs

Baseten Grid

Inference

Frontier Gateway

Training

Sandboxes

Development

Deployment

Engines

Model Labs

Organization

Hosting options

Observability

Troubleshooting

Deploy and iterate

Prepare a Python Truss project

Start a development deployment

Test an edit

Re-attach to a development deployment

What gets live-patched

What requires a full redeploy

Limitations

Publish and validate an update

Deploy to production

Clean up

CLI reference: model push

CLI reference: model watch

Autoscaling

Environments

​Prepare a Python Truss project

​Start a development deployment

​Test an edit

​Re-attach to a development deployment

​What gets live-patched

​What requires a full redeploy

​Limitations

​Publish and validate an update

​Deploy to production

​Clean up

CLI reference: model push

CLI reference: model watch

Autoscaling

Environments

Prepare a Python Truss project

Start a development deployment

Test an edit

Re-attach to a development deployment

What gets live-patched

What requires a full redeploy

Limitations

Publish and validate an update

Deploy to production

Clean up