# Connect your coding agent Source: https://docs.baseten.co/agent-setup Install the Baseten skill and MCP servers so your agent can manage your Baseten workspace and search these docs. Install the Baseten skill and MCP servers so your coding agent can manage your Baseten workspace and search these docs. To connect Claude Code, Codex CLI, Pi, Droid, or OpenCode to Model APIs, see [Coding agents](/inference/model-apis/coding-agents). ## Install the agent toolkit **To install the skill and MCP servers**: 1. Create a dedicated key with management permissions in your [API key settings](https://app.baseten.co/settings/api_keys). A dedicated key lets you revoke agent access without affecting inference or other tools. 2. Set it in your shell so the installer can read it: ```bash macOS / Linux theme={"system"} export BASETEN_MCP_KEY=... ``` ```powershell Windows PowerShell theme={"system"} $env:BASETEN_MCP_KEY = "..." ``` ```cmd Windows cmd theme={"system"} set BASETEN_MCP_KEY=... ``` To persist the key, add the line to your shell profile (`~/.zshrc` or `~/.bashrc`). On Windows, run `setx BASETEN_MCP_KEY "..."`. 3. Run these commands in a shell that exports `BASETEN_MCP_KEY`. The `-g` flag makes the installation available to each detected agent, and `-y` skips confirmation prompts: ```bash macOS / Linux theme={"system"} npx skills add basetenlabs/baseten-skills -g -y npx add-mcp https://api.baseten.co/mcp -g -y --header "Authorization: Bearer ${BASETEN_MCP_KEY}" npx add-mcp https://docs.baseten.co/mcp -n baseten_docs -g -y ``` ```powershell Windows PowerShell theme={"system"} npx skills add basetenlabs/baseten-skills -g -y npx add-mcp https://api.baseten.co/mcp -g -y --header "Authorization: Bearer $env:BASETEN_MCP_KEY" npx add-mcp https://docs.baseten.co/mcp -n baseten_docs -g -y ``` ```cmd Windows cmd theme={"system"} npx skills add basetenlabs/baseten-skills -g -y npx add-mcp https://api.baseten.co/mcp -g -y --header "Authorization: Bearer %BASETEN_MCP_KEY%" npx add-mcp https://docs.baseten.co/mcp -n baseten_docs -g -y ``` 4. Restart your coding agent so it loads the skill and MCP configuration. Alternatively, ask your agent to install the toolkit. Paste this prompt: ```md theme={"system"} Install the Baseten agent toolkit globally by following the instructions at https://github.com/basetenlabs/baseten-skills: - Install the `baseten` skill. - Add the backend MCP server at https://api.baseten.co/mcp with the header "Authorization: Bearer $BASETEN_MCP_KEY". - Add the docs MCP server at https://docs.baseten.co/mcp. Run the commands in a shell that exports BASETEN_MCP_KEY. Don't print the key. After installation, tell me how to verify the setup and whether I need to restart my agent. ``` **To confirm that the servers are connected**: Run `/mcp`. You should see: ```output theme={"system"} baseten ✔ connected baseten_docs ✔ connected ``` If Claude Code prompts before reading skill files, add this to `~/.claude/settings.json`: ```json theme={"system"} { "permissions": { "allow": ["Read(~/.claude/skills/**)"] } } ``` **To confirm that the servers are connected**: Open MCP settings and confirm both `baseten` and `baseten_docs` show as connected. Verify the setup by asking your agent to list the models in your workspace. If your agent exposes skills as slash commands, you can also run `/baseten`. The backend MCP server connects to the workspace associated with its API key. To work with multiple workspaces, install additional backend server instances under different names and use a different key for each one. ## Configure your agent manually Use these instructions if the installer does not detect your agent. Add both MCP servers to your agent's configuration, then install the skill with `npx skills add basetenlabs/baseten-skills`. For a docs-only setup, add `baseten_docs` without an authorization header. **To add the MCP servers by hand**: Add this to `mcp.json`: ```json mcp.json theme={"system"} { "mcpServers": { "baseten": { "type": "http", "url": "https://api.baseten.co/mcp", "headers": { "Authorization": "Bearer ${BASETEN_MCP_KEY}" } }, "baseten_docs": { "type": "http", "url": "https://docs.baseten.co/mcp" } } } ``` **To add the MCP servers by hand**: Add this to `.vscode/mcp.json`: ```json .vscode/mcp.json theme={"system"} { "servers": { "baseten": { "type": "http", "url": "https://api.baseten.co/mcp", "headers": { "Authorization": "Bearer ${BASETEN_MCP_KEY}" } }, "baseten_docs": { "type": "http", "url": "https://docs.baseten.co/mcp" } } } ``` **To add the MCP servers by hand**: 1. Install the skill: ```bash theme={"system"} npx skills add basetenlabs/baseten-skills ``` 2. Add the backend MCP server: ```bash theme={"system"} claude mcp add --transport http baseten https://api.baseten.co/mcp --header "Authorization: Bearer ${BASETEN_MCP_KEY}" ``` 3. Add the docs MCP server: ```bash theme={"system"} claude mcp add --transport http baseten_docs https://docs.baseten.co/mcp ``` `npx add-mcp` and `npx skills add` also detect Codex, Antigravity, Goose, Windsurf, and other supported agents. For GUI clients such as Claude Desktop, add both server URLs in the connector settings. Any MCP-compatible agent can use the URLs above. ## Next steps Call Model APIs from Pi, Droid, or OpenCode, or install Baseten Switch for Claude Code, Codex CLI, or Pi. Deploy a Hugging Face model with vLLM and the Baseten CLI. Append `.md` to any docs URL to open the page as Markdown. You can also use [llms.txt](https://docs.baseten.co/llms.txt) to index the documentation or [llms-full.txt](https://docs.baseten.co/llms-full.txt) to retrieve its full text. The **Copy page** button in the upper-right corner of each page copies the Markdown or opens it as plain text: The page context menu with options to copy or open documentation as Markdown. If you cannot install the full `baseten` skill, install the lightweight docs skill instead: ```bash theme={"system"} npx skills add https://docs.baseten.co ``` # Cancel a queued async request. Source: https://docs.baseten.co/api-reference/cancel-a-queued-async-request /reference/inference-api/inference-api-spec.json delete /async_request/{request_id} Cancels an async request. Only requests with `QUEUED` status may be canceled. Rate limited to 20 requests per second. # Get the status of an async request. Source: https://docs.baseten.co/api-reference/get-the-status-of-an-async-request /reference/inference-api/inference-api-spec.json get /async_request/{request_id} Returns the current status of an async model or chain request. Rate limited to 20 requests per second. # Asynchronously call a named environment of a chain. Source: https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-named-environment-of-a-chain /reference/inference-api/inference-api-spec.json post /environments/{env_name}/async_run_remote # Asynchronously call a named environment of a model. Source: https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-named-environment-of-a-model /reference/inference-api/inference-api-spec.json post /environments/{env_name}/async_predict # Asynchronously call a specific deployment of a chain. Source: https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-specific-deployment-of-a-chain /reference/inference-api/inference-api-spec.json post /deployment/{deployment_id}/async_run_remote # Asynchronously call a specific deployment of a model. Source: https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-specific-deployment-of-a-model /reference/inference-api/inference-api-spec.json post /deployment/{deployment_id}/async_predict # Asynchronously call the development deployment of a chain. Source: https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-development-deployment-of-a-chain /reference/inference-api/inference-api-spec.json post /development/async_run_remote # Asynchronously call the development deployment of a model. Source: https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-development-deployment-of-a-model /reference/inference-api/inference-api-spec.json post /development/async_predict # Asynchronously call the production environment of a chain. Source: https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-production-environment-of-a-chain /reference/inference-api/inference-api-spec.json post /production/async_run_remote Enqueues an asynchronous request for the chain deployment promoted to the production environment. # Asynchronously call the production environment of a model. Source: https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-production-environment-of-a-model /reference/inference-api/inference-api-spec.json post /production/async_predict Enqueues an asynchronous predict request for the deployment promoted to the production environment. Returns a request ID that can be used to poll for status or cancel the request. # Call a specific chain deployment by deployment ID. Source: https://docs.baseten.co/api-reference/non-regional/call-a-specific-chain-deployment-by-deployment-id /reference/inference-api/inference-api-spec.json post /deployment/{deployment_id}/run_remote # Call a specific deployment of a model by deployment ID. Source: https://docs.baseten.co/api-reference/non-regional/call-a-specific-deployment-of-a-model-by-deployment-id /reference/inference-api/inference-api-spec.json post /deployment/{deployment_id}/predict Sends a synchronous predict request to the specified deployment. # Call the chain deployment associated with a specified environment. Source: https://docs.baseten.co/api-reference/non-regional/call-the-chain-deployment-associated-with-a-specified-environment /reference/inference-api/inference-api-spec.json post /environments/{env_name}/run_remote # Call the development deployment of a chain. Source: https://docs.baseten.co/api-reference/non-regional/call-the-development-deployment-of-a-chain /reference/inference-api/inference-api-spec.json post /development/run_remote # Call the development deployment of a model. Source: https://docs.baseten.co/api-reference/non-regional/call-the-development-deployment-of-a-model /reference/inference-api/inference-api-spec.json post /development/predict Sends a synchronous predict request to the development deployment. # Call the model deployment associated with a specified environment. Source: https://docs.baseten.co/api-reference/non-regional/call-the-model-deployment-associated-with-a-specified-environment /reference/inference-api/inference-api-spec.json post /environments/{env_name}/predict Sends a synchronous predict request to the deployment promoted to the specified environment. # Call the production environment of a chain. Source: https://docs.baseten.co/api-reference/non-regional/call-the-production-environment-of-a-chain /reference/inference-api/inference-api-spec.json post /production/run_remote Sends a synchronous request to the chain deployment promoted to the production environment. The request body is forwarded to the chain's `run_remote` entrypoint. # Call the production environment of a model. Source: https://docs.baseten.co/api-reference/non-regional/call-the-production-environment-of-a-model /reference/inference-api/inference-api-spec.json post /production/predict Sends a synchronous predict request to the deployment promoted to the production environment. The request body is forwarded directly to the model's `predict` function. # Get async queue status for a named environment. Source: https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-a-named-environment /reference/inference-api/inference-api-spec.json get /environments/{env_name}/async_queue_status # Get async queue status for a specific deployment. Source: https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-a-specific-deployment /reference/inference-api/inference-api-spec.json get /deployment/{deployment_id}/async_queue_status # Get async queue status for the development deployment. Source: https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-the-development-deployment /reference/inference-api/inference-api-spec.json get /development/async_queue_status # Get async queue status for the production environment. Source: https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-the-production-environment /reference/inference-api/inference-api-spec.json get /production/async_queue_status Returns the number of queued and in-progress async requests for the deployment promoted to the production environment. Rate limited to 20 requests per second. # Wake a named environment of a model. Source: https://docs.baseten.co/api-reference/non-regional/wake-a-named-environment-of-a-model /reference/inference-api/inference-api-spec.json post /environments/{env_name}/wake # Wake a specific deployment of a model by deployment ID. Source: https://docs.baseten.co/api-reference/non-regional/wake-a-specific-deployment-of-a-model-by-deployment-id /reference/inference-api/inference-api-spec.json post /deployment/{deployment_id}/wake # Wake the development deployment of a model. Source: https://docs.baseten.co/api-reference/non-regional/wake-the-development-deployment-of-a-model /reference/inference-api/inference-api-spec.json post /development/wake # Wake the production environment of a model. Source: https://docs.baseten.co/api-reference/non-regional/wake-the-production-environment-of-a-model /reference/inference-api/inference-api-spec.json post /production/wake Triggers a wake for the deployment promoted to the production environment. Returns immediately with 202 Accepted. # Asynchronously call a regional environment of a chain. Source: https://docs.baseten.co/api-reference/regional/asynchronously-call-a-regional-environment-of-a-chain /reference/inference-api/inference-api-spec.json post /async_run_remote Enqueues an asynchronous run_remote request via a regional hostname. The environment is determined by the hostname, not the path. # Asynchronously call a regional environment of a model. Source: https://docs.baseten.co/api-reference/regional/asynchronously-call-a-regional-environment-of-a-model /reference/inference-api/inference-api-spec.json post /async_predict Enqueues an asynchronous predict request via a regional hostname. The environment is determined by the hostname, not the path. # Call a regional environment of a chain. Source: https://docs.baseten.co/api-reference/regional/call-a-regional-environment-of-a-chain /reference/inference-api/inference-api-spec.json post /run_remote Sends a synchronous run_remote request via a regional hostname. The environment is determined by the hostname, not the path. # Call a regional environment of a model. Source: https://docs.baseten.co/api-reference/regional/call-a-regional-environment-of-a-model /reference/inference-api/inference-api-spec.json post /predict Sends a synchronous predict request via a regional hostname. The environment is determined by the hostname, not the path. # Get async queue status for a regional environment. Source: https://docs.baseten.co/api-reference/regional/get-async-queue-status-for-a-regional-environment /reference/inference-api/inference-api-spec.json get /async_queue_status # Wake a regional environment of a model. Source: https://docs.baseten.co/api-reference/regional/wake-a-regional-environment-of-a-model /reference/inference-api/inference-api-spec.json post /wake # How Baseten works Source: https://docs.baseten.co/concepts/howbasetenworks The moving parts behind training, deployment, request routing, autoscaling, and environment promotion on Baseten. You can deploy an existing model or train a new one and deploy its checkpoint. This page explains the systems behind both paths: build pipelines, request routing, autoscaling, cold starts, and environments. For product options and starting points, see [Baseten overview](/overview). ## Multi-cloud Capacity Management (MCM) Multi-cloud Capacity Management (MCM) provisions and manages GPUs across cloud providers and geographic regions. When you request hardware, such as an H100 in US-East-1 or a cluster of B200s in a private region, MCM provisions it, configures networking, and monitors its health. Baseten provides a consistent training and inference runtime across the underlying infrastructure. MCM also supports high availability. Deployments run active-active across clusters and clouds. If a region or provider loses capacity, MCM reroutes and reprovisions workloads. ## Deploy an existing model To deploy a model, package it with [Truss](https://pypi.org/project/truss/), Baseten's open-source model packaging tool. Describe the model in a `config.yaml` (for supported architectures) or a small Python `Model` class (for custom code), then run `baseten model push` to ship it. `baseten model push` validates your `config.yaml`, archives your project directory, and uploads it to cloud storage. Baseten receives the archive and starts the build. For [Engine-Builder-LLM](/engines/engine-builder-llm/overview), Baseten downloads model weights from the source repository (Hugging Face, S3, or GCS) and compiles them with TensorRT-LLM. Compilation builds optimized CUDA kernels for the target GPU architecture, applies quantization if configured, and sets up tensor parallelism across multiple GPUs. Baseten packages the compiled engine, runtime configuration, and serving infrastructure into a container, deploys it to GPU infrastructure, and exposes it as an API endpoint. `baseten model push` returns once the upload finishes. For engine-based deployments, compilation can take several minutes. Watch progress in the deployment logs, or wait for the dashboard to show "Active." For [custom model code](/development/model/model-class), Baseten installs your Python dependencies, packages the `Model` class into a container, and deploys it. Custom builds do not apply Baseten engine optimizations automatically. Each push produces a container image identified by a content hash and stored in Baseten's container registry. The image is immutable, and an unchanged project reuses the cached image instead of triggering a new build. ## Train a model Use [Loops](/loops/overview) to write a Python training loop that calls a dedicated trainer and sampler. Use [Training Jobs](/training/overview) to run your own training container to completion. To run a training job, define the job in a Python configuration file (typically `config.py`) using the [`truss_train` SDK](/reference/sdk/training), then submit it with `baseten train push --config config.py`. Baseten provisions GPUs through MCM, runs your training container, and syncs checkpoints to storage as the job progresses. `baseten train push --config config.py` packages your training config, uploads it to Baseten, and starts the job on the hardware you specified (H100 or H200, single-node or multi-node). Your training code can use Axolotl, TRL, VeRL, Megatron, or any other framework you bundle into the container. Baseten runs your training container on the provisioned GPUs. As your training code writes checkpoints to the configured directory, Baseten uploads them to durable storage. If the job fails or you stop it, you can still use the most recent checkpoint. `baseten train checkpoint deploy --job-id ` constructs a Truss `config.yaml` from the checkpoint, packages it as a deployment, and exposes an API endpoint. From there, the deployment behaves like any other model on Baseten. For more information, see the [Training Jobs overview](/training/overview). ## Request routing Each model gets a dedicated subdomain: `https://model-{model_id}.api.baseten.co/`. The URL path selects the deployment that handles a request. `/production/predict` targets the production environment, and `/development/predict` targets the development deployment. You can also target a deployment by ID or an environment by name. Baseten resolves the target from the URL and routes the request to an active replica. If the deployment has scaled to zero, Baseten starts a replica and holds the request until the model loads. The request uses the same endpoint whether the replica is warm or cold-started. Engine-based deployments expose an [OpenAI-compatible API](/reference/inference-api/chat-completions) at `/v1/chat/completions`. Use the OpenAI SDK with the deployment's base URL and a Baseten API key. Custom model deployments use the [predict API](/reference/inference-api/overview), which accepts and returns arbitrary JSON. For long-running workloads, [async requests](/inference/async) return a request ID immediately. An async request service queues the request. A background worker then calls your model and delivers the result through a webhook. Sync requests get priority when capacity is tight, so background work doesn't starve real-time traffic. ## Autoscaling Baseten's autoscaler matches replica count to in-flight request load, keeping each replica below its [concurrency target](/deployment/autoscaling/overview). When average load over the autoscaling window (60 seconds by default) crosses the target utilization (70% by default), the autoscaler adds replicas up to the configured maximum. When load drops, the autoscaler waits for `scale_down_delay` (900 seconds by default), then removes excess replicas at a pace capped by `max_scale_down_rate` (half of running replicas by default). The timer resets after each reduction, and the cycle repeats until the deployment reaches its target size. This staged reduction prevents scaling changes in response to brief traffic dips. Set [`min_replica`](/deployment/autoscaling/overview) to 0 for scale-to-zero: the deployment incurs no GPU cost when idle, but the next request triggers a cold start. Set `min_replica` to 1 or higher to keep warm capacity ready, trading cost for lower latency. ## Cold starts and the Baseten Delivery Network Loading model weights can dominate cold-start time, especially when weights reach hundreds of gigabytes. The [Baseten Delivery Network (BDN)](/development/model/bdn) caches model weights across storage, clusters, and nodes. On the first deployment, BDN mirrors model weights from the source repository to Baseten storage. Later cold starts do not depend on the original Hugging Face, S3, or GCS source. When a replica starts, the BDN agent fetches a weight manifest, downloads files through a cache shared by the cluster, and stores them in a node-level cache. BDN deduplicates identical files across models, so a fine-tune that shares files with its base model downloads only the difference. Later cold starts on the same node or cluster can reuse cached weights. Container image streaming also lets the model begin loading weights before the image download completes. BDN serves training jobs the same way. Mount weights and training data into your training container from any supported source, and BDN caches them so subsequent jobs start faster. ## Environments and promotion Use a development deployment with scale-to-zero and live reload for fast iteration. When the model is ready for stable traffic, promote the development deployment directly to a named [environment](/deployment/environments), such as production or staging. Promotion creates a published deployment from the current development state. Each environment has its own stable URL, autoscaling settings, and metrics. When you promote a new deployment, Baseten applies the environment's autoscaling settings and routes its endpoint to the new deployment according to its promotion settings. The endpoint URL stays constant, so your calling code doesn't need to change. After the promotion, Baseten handles the previous deployment according to the environment's cleanup strategy. Promoting a published deployment reuses its existing image. Re-promoting a previous published deployment for rollback also reuses that image. Promoting a development deployment creates a new published deployment from its current state and triggers an image build. To skip the development stage, push directly to an environment with `baseten model push --environment staging`. Only one promotion can be active per environment at a time, which prevents conflicting updates. See [Deployment concepts](/deployment/concepts) for the full set of resource and CI/CD options. These pieces work the same whether you deploy an existing model or train a new one, so the path from prototype to production stays consistent. ## Next steps Deploy a Hugging Face model with `config.yaml` and the Baseten CLI. Run a fine-tune or pre-train and deploy the checkpoint to an endpoint. # Cold starts Source: https://docs.baseten.co/deployment/autoscaling/cold-starts Learn what makes a cold start slow and how to shrink it for your model. A *cold start* is the time a fresh replica spends starting up before it can accept traffic. By default, a synchronous request that triggers a cold start waits until the replica is ready, so startup time becomes part of the request's latency. The following diagram traces a deployment through that cycle, from **Scaled to zero** to **Active** and back, with the startup steps that add up to the wait. ## Cold start triggers Every new replica cold-starts before it can serve traffic, no matter why it was created. *Scale-from-zero* applies when a deployment's `min_replica` is 0. After traffic drops, the autoscaler waits for [`scale_down_delay`](/deployment/autoscaling/overview#param-scale-down-delay), removes replicas up to [`max_scale_down_rate`](/deployment/autoscaling/overview#param-max-scale-down-rate), and resets the delay. It repeats these steps until no replicas remain. A synchronous request can then trigger a replica startup and wait for it to finish. *Scaling events* happen while a deployment is already serving traffic. When load crosses the scaling threshold, the autoscaler adds replicas, and each one cold-starts before it can serve traffic. The replicas already running keep serving in the meantime, so users notice only when load grows faster than new replicas can start up. ## Contributing factors A new replica works through these steps in order, and their durations add up to the cold-start time: | Step | What happens | | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Container pull | The replica downloads your Docker image layers. | | Weight load | Model weights move from storage into accelerator memory. | | Engine initialization | Your model's setup code runs. For inference engines like vLLM and SGLang, this includes capturing CUDA graphs, compiling kernels with `torch.compile`, and profiling the KV cache. | Baseten provides the [Baseten Delivery Network (BDN)](/development/model/bdn), which speeds up weight load by mirroring your weights and caching them near your replicas. Each scale-up can read the weights from that cache instead of downloading them again from the source. Baseten also streams your container image in the background so image transfer can overlap other startup work. The dominant step depends on the model, runtime, and hardware. Engine initialization can dominate when graph capture or compilation is substantial. Weight loading can dominate when the model has larger weight files. Benchmark your deployment to identify the step to optimize. ## Reduce cold starts The biggest win comes from shrinking whichever step dominates startup. When that isn't enough, keep replicas warm so requests skip the cold start entirely. ### Faster weight loading BDN runs automatically on engine-builder deployments. On any other deployment, turn it on by adding a [`weights`](/development/model/bdn) block to your config. ### Compilation caching `torch.compile` creates artifacts while a replica starts. [Torch compile caching](/development/model/runtime-caching#torch-compile-caching), built on [b10cache](/development/model/runtime-caching), persists those artifacts so a new replica can reuse them instead of compiling from scratch. Benchmark your deployment to measure the effect on startup time. ### Warm replicas `min_replica` sets a floor on running replicas. Keep it at 1 or higher so a replica stays warm to serve the first request. You pay for that replica while it's idle, but requests avoid a scale-from-zero wait or rejection. Set it in the dashboard or through the [autoscaling settings API](/reference/management-api/deployments/autoscaling/updates-a-deployments-autoscaling-settings): ```json Autoscaling settings theme={"system"} { "min_replica": 1 } ``` For production redundancy, set `min_replica` to 2 or higher so another warm replica remains available while Baseten replaces a failed or restarting replica. Your replica floor trades cost against latency: | Approach | Cost | Latency | Best for | | -------------------------------- | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------- | ----------------------------------------------------- | | Scale to zero (`min_replica: 0`) | No running-replica charges while idle; wake-up minutes are [billed](/organization/billing) | Synchronous requests can wait for a replica to start. | Batch jobs, development, and spiky low-volume traffic | | Always on (`min_replica` ≥ 1) | Pay for idle replicas | No cold start from idle, though new replicas still cold-start | Latency-sensitive production traffic | Start warm for production, and scale to zero only when an occasional cold-start wait or retry is acceptable. ### Pre-warming For predictable traffic spikes, raise `min_replica` ahead of the expected load: ```bash Terminal theme={"system"} # Run before the spike. Use your observed cold start as the lead time. curl -X PATCH \ https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/autoscaling_settings \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{"min_replica": 5}' ``` After traffic stabilizes, reset to your normal minimum. ### Scale-down delay A longer scale-down delay keeps replicas warm through brief traffic dips. The default is 15 minutes (900 seconds); this example doubles it to 30 minutes: ```json Autoscaling settings theme={"system"} { "scale_down_delay": 1800 } ``` A replica that's still warm when traffic returns serves immediately, with no cold start. ## Next steps * [Request lifecycle](/deployment/autoscaling/request-lifecycle): What happens to requests during cold starts, including queuing and timeout behavior. * [Autoscaling](/deployment/autoscaling/overview): Configure `min_replica`, `scale_down_delay`, and the rest of the scaling settings. * [Traffic patterns](/deployment/autoscaling/traffic-patterns): Pre-warming strategies for different traffic types. * [Billing and usage](/organization/billing): How cold-start time is metered. * [Troubleshooting](/troubleshooting/deployments#autoscaling-issues): Diagnose cold start issues. # Autoscaling Source: https://docs.baseten.co/deployment/autoscaling/overview Configure autoscaling to dynamically adjust replicas based on traffic while minimizing idle compute costs. Autoscaling adjusts the number of **replicas** behind a deployment as request load changes. It adds replicas when traffic rises and removes them when traffic falls. Configure the minimum, maximum, and scaling thresholds to balance latency, capacity, and cost. Baseten [bills each running replica by the minute](/organization/billing). A deployment at zero replicas incurs no GPU charges, but starting and loading a new replica is billable. See [Cold starts](/deployment/autoscaling/cold-starts) for ways to reduce startup time. Start with the default settings, then tune them for your model and traffic pattern. | Parameter | Default | Range | What it controls | | ------------------- | ------- | -------- | -------------------------------------------- | | Min replicas | 0 | ≥ 0 | Baseline capacity (0 = scale to zero). | | Max replicas | 1 | ≥ 1 | Cost/capacity ceiling. | | Autoscaling window | 60s | 10-3600s | Time window for traffic analysis. | | Scale-down delay | 900s | 0-3600s | Wait time before removing idle replicas. | | Max scale-down rate | 50% | 1-50% | Cap on replicas removed per scale-down step. | | Concurrency target | 1 | ≥ 1 | Requests per replica before scaling. | | Target utilization | 70% | 1-100% | Headroom before scaling triggers. | Use this page to choose autoscaling settings and understand how they interact. To apply settings to a live deployment from the Baseten dashboard or Management API, see [Scale a deployment](/deployment/manage/scaling). To change settings on a recurring or one-time schedule, see [Schedule autoscaling](/deployment/autoscaling/schedules). ## How autoscaling works The autoscaler averages in-flight requests over `autoscaling_window` (60 seconds by default). It divides that load by each replica's effective capacity (`concurrency_target` × `target_utilization_percentage`) and rounds up to calculate the desired replica count. Scale-up begins at the next decision. Scale-down waits for `scale_down_delay` and removes at most `max_scale_down_rate` of the running replicas in each step. The slower scale-down prevents brief traffic dips from repeatedly stopping and starting replicas. Use the simulator to see how the settings respond to different traffic patterns. Start with a scenario that demonstrates a cold start or scaling oscillation, or adjust the traffic and parameters yourself. The meters show idle capacity and queued requests. The simulator models requests waiting for capacity, not error responses from load shedding. See [Request lifecycle](/deployment/autoscaling/request-lifecycle#request-queuing-and-load-shedding). For example, set `concurrency_target` to 10 and `target_utilization_percentage` to 70%. Each replica then has an effective capacity of 7 concurrent requests (10 × 0.70). At an average of 25 in-flight requests, the autoscaler calculates ⌈25 / 7⌉ = 4 desired replicas and starts the additional capacity. In the simulator, further requests queue after the deployment reaches `max_replica`. Compare the **Desired** line on the [Replicas graph](/observability/metrics#replicas) against your current replica count to confirm your settings produce the scale you expect. When average load drops below the threshold, the autoscaler waits for `scale_down_delay` (900 seconds by default), removes replicas up to `max_scale_down_rate` (50% by default), and resets the timer. At the default rate, eight replicas reduce to four, then two, then one, with a full delay between steps. If traffic returns during the delay, the replicas stay active. Scale-down stops at `min_replica`. ## Replicas Each replica is an independent instance of the model on its own hardware. Replicas serve requests in parallel. You set the minimum and maximum replica counts, and the autoscaler adjusts capacity within those boundaries. The floor for your deployment's capacity. The autoscaler won't scale below this number. **Range:** ≥ 0 The default of 0 enables *scale-to-zero*: when no requests arrive for long enough, all replicas shut down and your deployment incurs no running-replica charges. When traffic returns, a synchronous request can wait while the deployment [cold-starts](/deployment/autoscaling/cold-starts) a replica. During startup, [billing is per minute](/organization/billing) even though the replica isn't yet serving responses. Use `min_replica: 1` as a starting point when you need to avoid scale-from-zero latency. Use at least 2 when you also need replica redundancy. Replicas added during scale-up still cold-start. The ceiling for your deployment's capacity. The autoscaler won't scale above this number. **Range:** ≥ 1 This setting limits capacity and cost. When traffic exceeds the capacity of the maximum replicas, the autoscaler doesn't start more replicas. Synchronous requests can wait for an open slot, and load shedding can reject traffic if the queue grows too large. See [Request lifecycle](/deployment/autoscaling/request-lifecycle#request-queuing-and-load-shedding). The default of 1 caps the deployment at one replica. With the default `min_replica` of 0, the deployment can run zero or one replica, depending on demand. Estimate max replicas: $$ (peak\_requests\_per\_second / throughput\_per\_replica) + buffer $$ For high-volume workloads requiring guaranteed capacity, [contact Baseten](mailto:support@baseten.co) about reserved capacity options. ## Scaling triggers The autoscaler compares in-flight requests with a per-replica threshold. `concurrency_target` sets how many simultaneous requests a replica accepts. `target_utilization_percentage` reserves headroom by triggering scale-up before replicas reach that limit. Scale-up begins when: $$ load > replicas \times concurrency\_target \times target\_utilization $$ The following diagram shows a replica with `concurrency_target` of 8 and `target_utilization` of 50%, so the per-replica threshold sits at 4. The first four requests fill capacity within headroom; the fifth crosses the threshold, and the autoscaler provisions a second replica to absorb the overflow before the remaining slots saturate. How many requests each replica can handle simultaneously. This directly determines replica count for a given load. **Range:** ≥ 1 Given the current load, the autoscaler calculates desired replicas: $$ desired\_replicas = \lceil in\_flight\_requests / (concurrency\_target \times target\_utilization) \rceil $$ *In-flight requests* are requests sent to your model that haven't returned a response (for streaming, until the stream completes). [Async inference requests](/inference/async) are not included in this count. This count is exposed as [`baseten_concurrent_requests`](/observability/export-metrics/supported-metrics#baseten_concurrent_requests) in the metrics dashboard and metrics export. The right value depends on how your model uses hardware. Benchmark representative requests on your chosen instance type. A model that processes one request at a time may need a concurrency target of 1. Engines that batch requests may support higher targets, but setting the target beyond measured capacity adds per-replica queuing and latency. Higher concurrency uses fewer replicas but can increase per-replica queuing and latency. Lower concurrency uses more replicas but reduces queuing. **Starting points by model type:** Use these values to choose an initial benchmark. Your model, engine configuration, request shape, and hardware determine the final setting. | Model type | Starting concurrency | | ----------------------- | -------------------- | | Standard Truss model | 1 | | vLLM / LLM inference | 32-128 | | SGLang | 32 | | Text embeddings (TEI) | 32 | | BEI embeddings | 96+ (min ≥ 8) | | Whisper (async batch) | 256 | | Image generation (SDXL) | 1 | For engine-specific guidance, see [Autoscaling engines](/engines/performance-concepts/autoscaling-engines). **Concurrency target** controls requests sent *to* a replica and triggers autoscaling. **predict\_concurrency** (Truss config.yaml) controls requests processed *inside* the container. Concurrency target should be less than or equal to predict\_concurrency. See the `predict_concurrency` field in the [Truss configuration reference](/reference/truss-configuration) for details. Headroom before scaling triggers. The autoscaler scales when utilization reaches this percentage of the concurrency target, not when replicas are fully loaded. **Range:** 1-100% The effective threshold is: $$ concurrency\_target × target\_utilization $$ With a concurrency target of 10 and utilization of 70%, scaling triggers at 7 concurrent requests (10 × 0.70), leaving 30% headroom for absorbing spikes while new replicas start. As a starting point, lower values such as 50% to 60% provide more headroom for spikes but cost more. Higher values such as 80% or more use capacity more fully, but leave less headroom while new replicas start. Target utilization is **not** GPU utilization. It measures request slot usage relative to your concurrency target, not hardware utilization. ## Scaling dynamics Three settings control scaling speed. `autoscaling_window` determines how much traffic history each decision uses. `scale_down_delay` keeps replicas active for a period after load drops. `max_scale_down_rate` limits how many replicas each scale-down step removes. The diagram shows traffic falling to zero and a replica stopping after the delay expires. How far back (in seconds) the autoscaler looks when measuring traffic. Traffic is averaged over this window to make scaling decisions. **Range:** 10-3600 seconds A 60-second window smooths out momentary spikes by averaging load over the past minute. Start with 30 to 60 seconds for bursty traffic. Start with 2 to 5 minutes when you want to filter short-lived fluctuations. Adjust the window against your observed traffic and replica startup time. How long (in seconds) the autoscaler waits after load drops before removing replicas. **Range:** 0-3600 seconds When load drops, the autoscaler starts a countdown. If load stays low for the full delay, it removes replicas in steps, each capped by `max_scale_down_rate`, with a fresh delay between steps. If traffic returns before the countdown finishes, the replicas stay active and the countdown resets. If replicas repeatedly scale up and down, increase this value first. [BIS-LLM](/engines/bis-llm/overview) deployments default to 300 seconds. The maximum percentage of running replicas the autoscaler can remove in one scale-down step. **Range:** 1-50% Each time a `scale_down_delay` elapses, the autoscaler removes at most this percentage of running replicas. The default of 50% produces the halve-and-wait pattern described above. Lower values release capacity more gradually, which keeps more replicas warm when traffic tends to rebound shortly after it drops. BIS-LLM deployments default to 20% because replicas hold KV cache that's expensive to rebuild. A **short window** with a **long delay** gives you fast scale-up while maintaining capacity during temporary dips. This is a good starting configuration for most workloads. ## Development deployments Development deployments are designed for iteration, not production traffic. Replicas are fixed at 0-1 to match the [`truss watch`](/reference/cli/truss/watch) workflow, where you're testing changes on a single instance rather than handling concurrent users. You can still adjust timing and concurrency settings. | Setting | Value | Modifiable | | ------------------ | ----------- | ---------- | | Min replicas | 0 | No | | Max replicas | 1 | No | | Autoscaling window | 60 seconds | Yes | | Scale-down delay | 900 seconds | Yes | | Concurrency target | 1 | Yes | | Target utilization | 70% | Yes | To configure the full set of autoscaling settings, [promote the deployment to an environment](/deployment/manage/lifecycle#promote-to-an-environment). ## Troubleshooting See [Autoscaling troubleshooting](/troubleshooting/deployments#autoscaling-issues) for help with oscillation, slow scale-up, and unexpected costs. ## Next steps Identify your traffic pattern and get recommended starting settings. Understand cold starts and how to minimize their impact. Complete autoscaling API documentation. Recommended settings for BEI and Engine-Builder-LLM with dynamic batching. # Request lifecycle Source: https://docs.baseten.co/deployment/autoscaling/request-lifecycle How Dedicated Inference routes and queues requests, applies timeouts, and returns errors. This page covers requests to models and Chains that you deploy with Dedicated Inference. When you call an OpenAI-compatible endpoint or the [predict API](/inference/calling-your-model), Baseten authenticates and routes the request before selecting a replica. For a Truss model with custom code, the replica runs your `predict` function after these steps. The request path explains where latency occurs, what each status code means, and how to debug a failed request. ## How a request reaches your model Your request first hits Baseten's inference gateway, which authenticates it against your [API key](/organization/api-keys). If authentication fails, the gateway rejects the request before it reaches any model infrastructure: `401` when the request has no credentials, `403` when the key is invalid. A standard endpoint also returns `403` when the environment [requires a regional endpoint](/deployment/regional-environments#compare-regional-and-standard-endpoints). Once authenticated, the request moves to the routing layer. Baseten selects a workload plane where the deployment runs, then distributes traffic among the deployment's replicas in that plane. The [concurrency target](/deployment/autoscaling/overview#param-concurrency-target) defines how many simultaneous requests each replica should handle and informs autoscaling decisions. When routing succeeds, Baseten forwards the request to the deployment, and a replica runs inference. For deployments that use the predict API, your `predict` function executes here. The response flows back through the same path to the client. For most requests, the routing overhead is negligible compared to your model's inference time. The sections below cover what happens when this straightforward path breaks down: when no replica is available, when replicas reach capacity, and when requests fail partway through. ## What happens when no replica is available If your deployment has scaled to zero, no replica is ready to receive a request. By default, Baseten parks a synchronous request at the routing layer and waits for a replica to become available. Once a replica is ready, Baseten forwards the parked request, and the model processes it normally. From the client's perspective, this wait adds to the normal inference time. This parking behavior makes [scale-to-zero](/deployment/manage/scaling#scale-to-zero) practical for deployments that queue requests. You don't need to retry a request because your deployment was idle. The request waits for a replica, but only until the parking timeout expires. The parking timeout uses the same configured duration as the predict timeout, which is 1200 seconds by default. If no replica becomes available by then, the request fails with a `500`. To avoid scale-from-zero latency, keep [minimum replicas](/deployment/autoscaling/overview#param-min-replica) greater than zero. [Async requests](/inference/async) follow a different path. The `/async_predict` endpoint accepts the request into the async service without waiting for a replica. The service dispatches the request later and applies its own [inference retry policy](/inference/async#inference-retries). For strategies to reduce cold start latency, including warm replicas, pre-warming, and the Baseten Delivery Network, see [Cold starts](/deployment/autoscaling/cold-starts). ## Request queuing and load shedding Even when replicas are running, they can fill up. By default, incoming synchronous requests wait for an open replica slot. The autoscaler can add replicas within your configured limits. Load shedding prevents the queue from growing without bound. Baseten rejects a new request with a `429` when queued payloads create memory pressure or the queue crosses its soft limit. Baseten returns a `529` when the queue crosses its hard limit. Retry either response with exponential backoff and jitter. Your client can't distinguish time spent in the queue from time spent running inference. Set a client-side timeout that matches your latency requirements. If queuing persists, increase [max replicas](/deployment/autoscaling/overview#param-max-replica) when the deployment reaches its replica ceiling. Raise the [concurrency target](/deployment/autoscaling/overview#param-concurrency-target) only when benchmarks show that each replica can handle more simultaneous requests. ### Request backpressure policies A request backpressure policy controls what happens to a new synchronous request when no replica has an open request slot: | Policy | Behavior | | ------------------ | ------------------------------------------------------------------------------ | | **Queue on full** | Waits for an open replica slot. This is the default. | | **Reject on full** | Returns `529` immediately with the error code `request_backpressure_rejected`. | Use **Queue on full** when your clients can tolerate variable latency and you want the deployment to absorb short traffic spikes. Use **Reject on full** when clients have strict latency limits or can retry requests elsewhere. Rejected requests don't reach a replica, so they don't contribute to concurrency-based autoscaling. Set the [target utilization percentage](/deployment/autoscaling/overview#param-target-utilization) below 100% so admitted traffic can trigger scaling before every request slot fills. If the deployment has scaled to zero, the first request triggers a replica start in the background. That request and later requests receive `529` until a replica is ready. Retry the request with exponential backoff and jitter. The reject policy applies only to synchronous HTTP requests. It doesn't change how `/async_predict` queues and retries inference. Request backpressure policies aren't available for WebSocket, gRPC, Chain, or [Baseten Inference Stack](/engines/bis-llm/overview) deployments. The **Request handling** setting doesn't affect Baseten Inference Stack admission behavior. Baseten Inference Stack uses separate router-level capacity controls, which can return `429` responses based on request shape, such as the number of cached tokens and tokens that aren't cached. Policy rejections appear as `5xx` responses in the [Inference volume](/observability/metrics#inference-volume) chart. They also appear in [deployment logs](/observability/logs#scope-by-environment-or-deployment) with the error code `request_backpressure_rejected`. ### Backpressure configuration Request backpressure policies are rolling out gradually. If the **Request backpressure policy** setting isn't visible in your deployment or environment settings, [contact support](mailto:support@baseten.co) to enable it for your organization. Choose a standalone deployment to configure only that deployment. Choose an environment to keep the policy with the environment as you promote new deployments into it. You can't change the policy while a rolling deployment is in progress. Wait for the promotion to finish, or cancel the promotion before changing the policy. **To configure a request backpressure policy**: ```bash Command theme={"system"} baseten model deployment update-request-backpressure \ --model-id \ --deployment-id \ --policy reject-on-full ``` ```txt Output theme={"system"} Request backpressure policy: reject-on-full ``` To restore queuing, pass `--policy queue-on-full`. Pass `--policy null` to clear an explicit policy and use the default. For more information, see [`deployment update-request-backpressure`](/reference/cli/baseten/model-deployment#update-request-backpressure). **To configure a request backpressure policy**: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar. 2. Select your model. 3. Under **Deployments** or **Environments**, select the deployment or environment you want to configure. 4. Under **Request handling**, choose **Manage**. 5. For **Request backpressure policy**, select **Queue on full (default)** or **Reject on full**. 6. Choose **Update**. **To update a standalone deployment's request backpressure policy**: 1. Get the model ID and deployment ID you want to configure. 2. Update the deployment's request backpressure settings: ```bash Request theme={"system"} curl -X PATCH "https://api.baseten.co/v1/models/abc123/deployments/def456/request_backpressure_settings" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -H "Content-Type: application/json" \ -d '{"policy": "REJECT_ON_FULL"}' ``` ```json Response theme={"system"} { "policy": "REJECT_ON_FULL" } ``` 3. Confirm that the response contains `"policy": "REJECT_ON_FULL"`. **To update an environment's request backpressure policy**: This procedure requires the external `jq` command to display only the request backpressure settings from the response. 1. Get the model ID and environment name you want to configure. 2. Update the environment settings: ```bash Request theme={"system"} curl -X PATCH "https://api.baseten.co/v1/models/abc123/environments/production" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -H "Content-Type: application/json" \ -d '{"request_backpressure_settings": {"policy": "REJECT_ON_FULL"}}' \ | jq '.environment.request_backpressure_settings' ``` ```json Response theme={"system"} { "policy": "REJECT_ON_FULL" } ``` 3. Confirm that the response contains `"policy": "REJECT_ON_FULL"`. **To set a request backpressure policy when you create an environment**: This procedure requires the external `jq` command to display only the request backpressure settings from the response. 1. Get the model ID for the new environment. 2. Include `request_backpressure_settings` in the create request: ```bash Request theme={"system"} curl -X POST "https://api.baseten.co/v1/models/abc123/environments" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -H "Content-Type: application/json" \ -d '{"name": "staging", "request_backpressure_settings": {"policy": "REJECT_ON_FULL"}}' \ | jq '.request_backpressure_settings' ``` ```json Response theme={"system"} { "policy": "REJECT_ON_FULL" } ``` 3. Confirm that the response contains `"policy": "REJECT_ON_FULL"`. To restore queuing for a standalone deployment, send `{"policy": "QUEUE_ON_FULL"}`. For an environment, send `{"request_backpressure_settings": {"policy": "QUEUE_ON_FULL"}}`. In either payload, replace `"QUEUE_ON_FULL"` with `null` to clear an explicit policy and use the default effective policy. ## Internal retries When a replica returns a retryable status such as `502`, `503`, or `504`, the routing layer retries the request with exponential backoff. Connection failures use the same retry path. Retries stop when they reach the configured maximum number of prediction attempts or the request deadline, whichever comes first. The routing layer doesn't apply these retries to async requests. The async service has a separate [inference retry policy](/inference/async#inference-retries). From your client's perspective, retries add latency instead of immediately returning an error. Check the `X-BASETEN-MODEL-PREDICTION-ATTEMPTS` response header when a request takes longer than expected. A value greater than 1 confirms that Baseten retried the request. Baseten can temporarily suppress retries when the routing layer is under pressure. If a sticky-session request returns a `503`, the retry routes to a different replica. ## Timeouts The **predict timeout** controls how long a sync request can take from the moment it's forwarded to a replica until a response must be returned. If your model's inference exceeds this window, the request is cancelled and the client receives a `504`. The server-side default is 1200 seconds (20 minutes). If you need requests to fail faster than that, set a client-side timeout in your HTTP client. The **async predict timeout** is 3600 seconds (1 hour) per inference attempt. A timeout produces a retryable `504` within the async service. Baseten follows the request's [async inference retry policy](/inference/async#inference-retries). After the service exhausts those attempts, it marks the request as failed with a `MODEL_PREDICT_TIMEOUT` error status and sends the error payload to your webhook. For a parked synchronous request, the **parking timeout** controls how long it waits when the deployment has no ready replicas. It uses the same configured duration as the synchronous predict timeout, but the two timeouts apply separately. If routing succeeds before the parking timeout expires, the full predict timeout starts when Baseten forwards the request to a replica. With the default settings, a synchronous request can spend up to 1200 seconds parked and then up to another 1200 seconds running inference. For **streaming responses**, timeouts behave differently because the HTTP headers, including the `200` status code, are sent when the stream begins. If the timeout expires mid-stream, the stream stops and the connection closes without an error code, since the status was already written. Most HTTP clients surface this as a connection reset or incomplete response rather than a timeout error. ## Requests during a promotion You can [promote a deployment](/deployment/manage/lifecycle#promote-to-an-environment) while the current one is serving live traffic without dropping in-flight requests. As traffic shifts to the new deployment, each replica of the previous one stops accepting new requests and finishes the work it's already doing before shutting down. There's nothing to configure: draining happens automatically, whether the promotion replaces the deployment immediately or through a [rolling deployment](/deployment/rolling-deployments). Draining replicas stay alive for up to one hour. This window covers a single in-flight attempt: synchronous prediction has a 1200-second default timeout, and an async prediction attempt has a 3600-second timeout. The async service schedules any retry as a separate attempt. If your model runs its own shutdown logic, such as flushing state in a custom server, it receives a `SIGTERM` when its replica starts draining. ## HTTP status codes This table summarizes the status codes produced by the routing and lifecycle behavior on this page. Request-side errors such as `400` and `413` come from other layers. For what each error means, how to tell a model failure from a Baseten-side issue, and where to look next, see [Inference errors](/inference/errors). | Code | Meaning | When it occurs | What to do | | ------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `200` | Success | Normal predict response. | None. | | `201` | Created | Async predict request queued successfully. | Use a [webhook](/inference/async#webhooks) to receive the output. Poll the status endpoint only to monitor the request lifecycle. | | `401` / `403` | Unauthorized / Forbidden | Missing credentials (`401`), an invalid API key (`403`), or use of the standard endpoint for an environment that requires a regional endpoint (`403`). | Check your [API key](/organization/api-keys). For a regional environment, use its [regional endpoint](/deployment/regional-environments#regional-endpoint-formats). | | `429` | Too Many Requests | No replica slot was available (`CAPACITY_EXCEEDED`), or load shedding detected memory pressure or the soft queue limit. | Retry with exponential backoff. If persistent, increase [max replicas](/deployment/autoscaling/overview#param-max-replica) or tune the [concurrency target](/deployment/autoscaling/overview#param-concurrency-target). | | `499` | Client Closed Request | Client disconnected before the response was written. | No server-side action needed. Review client-side timeout configuration if unexpected. | | `500` | Internal Server Error | A sync request's parking timeout expired before a replica became available. | Retry with exponential backoff. If persistent, keep [minimum replicas](/deployment/autoscaling/overview#param-min-replica) greater than zero or reduce [cold-start time](/deployment/autoscaling/cold-starts). | | `502` | Bad Gateway | The model became unavailable during inference. | Retry. If persistent, check model logs for crashes or errors in your `predict` function. | | `503` | Service Unavailable | The routing layer couldn't find a replica endpoint, typically during a deployment rollout or immediately after a replica failure. | Retry. If persistent, check deployment status in the Baseten dashboard. | | `504` | Gateway Timeout | The request exceeded the server-side predict timeout (1200 seconds). | Optimize your model's inference speed or use [async inference](/inference/async) for longer work. | | `529` | Overloaded | The **Reject on full** policy rejected the request because no replica slot was available, or load shedding detected that the queue crossed its hard limit. Policy rejections include the error code `request_backpressure_rejected`. | Retry with exponential backoff and jitter. If persistent, increase [max replicas](/deployment/autoscaling/overview#param-max-replica), tune the [concurrency target](/deployment/autoscaling/overview#param-concurrency-target), or use **Queue on full** when clients can tolerate the added latency. | A `500` from a sync request during a cold start can mean the parking timeout expired before a replica finished starting. Retry with exponential backoff. If this recurs, keep [minimum replicas](/deployment/autoscaling/overview#param-min-replica) greater than zero or reduce [cold-start time](/deployment/autoscaling/cold-starts). ## Request cancellation When a client disconnects before the response is written, the routing layer detects the closed connection and cancels the in-flight work. The server logs this as a `499`. In the common case, such as a user closing a browser tab or a client-side timeout firing, this is harmless and the `499` is informational rather than an error. The more important question is whether cancellation propagates all the way to the GPU. If a client disconnects during a long generation and the model keeps running, you're paying for GPU time that produces tokens nobody will read. Baseten cancels in-flight work automatically so this doesn't happen. When the routing layer detects a disconnect, it signals the inference engine, which aborts the running request and frees GPU resources. This works across engines including TRT-LLM and vLLM. If you're using a custom model server, you can implement cancellation yourself using Truss request objects. See [Request handling](/development/model/streaming-and-endpoints#request-handling) for code examples. ## Next steps Reduce cold start latency with warm replicas and pre-warming strategies. Configure concurrency targets, replica counts, and scaling dynamics. Fire-and-forget inference with webhook delivery. Diagnose common deployment issues including autoscaling problems. # Schedule autoscaling for traffic Source: https://docs.baseten.co/deployment/autoscaling/schedules Change autoscaling settings on a recurring or one-time window instead of running an external cron job. Autoscaling responds to traffic that has already arrived, so requests at the start of an increase can wait behind a [cold start](/deployment/autoscaling/cold-starts) while new replicas start. When your traffic follows known clock times, you can change the autoscaling settings before the traffic arrives rather than after. An *autoscaling schedule* is a window on a model [environment](/deployment/environments), recurring or one-time, carrying its own autoscaling settings. While the window is active, the environment runs the schedule's settings. Outside every window, it runs the environment's default configuration. Schedules do this natively, so you don't need an external [cron job](/deployment/autoscaling/traffic-patterns#scheduled-pre-warming) or [GitHub Action](/deployment/ci-cd) calling the [Management API](/deployment/manage/scaling) on a timer. ## How schedules work Autoscaling defaults to the configuration values you set on the environment. To vary those values by time of day or day of week, define up to 10 schedules, each pairing a recurring window with its own set of autoscaling settings. The schedule whose window covers the current time is the one in effect. A schedule's window runs on one of three cadences: * **Daily**: one window a day, between a start and end time, on the days you choose, such as `09:00` to `18:00` on Monday through Friday. * **Hourly**: a window in every hour, optionally limited to a range of hours, such as `:55` to `:20` during hours `09` to `17`. * **One time**: a single window between two absolute date-times, such as September 1 at `09:00` to `13:00`. Use one-time schedules for planned events like launches, load tests, and migrations. The window does not repeat, and the schedule stops applying when it ends. Windows include their start and exclude their end: in a `09:00` to `17:00` window, `09:00` is inside and `17:00` is not. A recurring end time at or before the start wraps past midnight, so a `22:00` to `06:00` window covers the overnight hours. Every schedule on an environment shares one timezone, set once for the whole set. Recurring times are local to it; one-time windows accept any timezone offset and convert to UTC. ## Configure a schedule You configure schedules per environment, from the same dialog as the rest of your autoscaling settings. **To add an autoscaling schedule**: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar. 2. Select your model, then select the deployment for the environment you want to schedule. 3. Choose **Configure autoscaling**. 4. For **Config to edit**, choose **Schedules**. 5. Choose **Add schedule**, then set its window and replicas: * For **Frequency**, choose **Daily**, **Hourly**, or **One time**. * For a daily or hourly schedule, select the days it runs under **On days**, then set **Starts at** and **Ends at**. For an hourly schedule, use **Limit to hours** to restrict which hours it runs in. * For a one-time schedule, set the **Schedule window** start and end date-times. * Set **Min** and **Max** replicas for the window. To override the remaining autoscaling settings, choose **More options**. 6. Choose **Save**. If the new schedule's window covers the current time, the environment switches to it as soon as you save. An **Active** badge marks whichever configuration is running. A start time is the earliest the settings change, not a promise that replicas are ready to serve. If the deployment can't take the change right then, because it's deploying, loading a model, mid [rolling promotion](/deployment/rolling-deployments), or migrating, Baseten applies it as soon as it can. ## Validation Saving fails if any schedule breaks one of these rules: * **Count**: up to 10 schedules per environment. Expired one-time schedules don't count. * **Window length**: at least 5 minutes, long enough to scale up and back down. A one-time window can span at most 48 hours, including back-to-back chains of one-time schedules. * **Overlap**: two windows can't cover the same time. * **Gaps**: windows either touch exactly or leave at least 5 minutes between them. * **Coverage**: schedules can't fill the entire week, so the default configuration always has time to run. * **Timezone**: one shared timezone per environment. * **One-time start**: a new one-time schedule must start in the future. You can still edit an existing one-time schedule after its window starts. * **One-time horizon**: a one-time window can end at most 90 days in the future. Windows that touch exactly hand off directly. A `05:40` to `09:00` window can hand straight to a `09:00` to `18:00` window with no gap and no return to the default in between. ## Edit or remove a schedule Changes to the schedule that's currently running apply immediately, not at its next start. Disabling or removing it falls back to the default, or to another schedule whose window covers the current time. **To edit, rename, duplicate, disable, or remove a schedule**: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar. 2. Select your model, then select the deployment for the environment. 3. Choose **Configure autoscaling**. 4. For **Config to edit**, choose **Schedules**. 5. Expand the schedule and make your changes: * To rename, duplicate, or remove it, use the actions menu on its header. * To pause it without removing it, use the **Enabled** switch. 6. Choose **Save**. A disabled schedule still has to pass validation, so you can re-enable it later. A one-time schedule stays on the environment after its window passes; remove it when you no longer need it. ## Manage schedules from the Baseten CLI The [`baseten model environment autoscaling-schedule`](/reference/cli/baseten/model-environment-autoscaling-schedule) commands manage the same schedule collection as the dashboard and API. ### Create a schedule ```bash Command theme={"system"} baseten model environment autoscaling-schedule create \ --model-id \ --environment production \ --name "Weekday business hours" \ --cadence daily \ --weekdays monday,tuesday,wednesday,thursday,friday \ --start-hour 9 \ --end-hour 18 \ --min-replica 2 \ --max-replica 8 \ --timezone America/Los_Angeles ``` ```txt Output theme={"system"} Created autoscaling schedule Weekday business hours ``` For a one-time schedule, use `--cadence one-time` with `--start-at` and `--end-at` instead of the weekday and hour flags. The `--cadence` flag accepts `daily`, `hourly`, or `one-time`. For the full flag set, see [`autoscaling-schedule create`](/reference/cli/baseten/model-environment-autoscaling-schedule#create). ### Update a schedule ```bash Command theme={"system"} baseten model environment autoscaling-schedule update \ --model-id \ --environment production \ --schedule-name "Weekday business hours" \ --max-replica 12 ``` Target a schedule by `--schedule-id` or `--schedule-name`. Changing `--cadence` requires the new shape's timing flags, since the old timing doesn't carry over. For the full flag set, see [`autoscaling-schedule update`](/reference/cli/baseten/model-environment-autoscaling-schedule#update). ### Delete a schedule ```bash Command theme={"system"} baseten model environment autoscaling-schedule delete \ --model-id \ --environment production \ --schedule-name "Weekday business hours" ``` ```txt Output theme={"system"} Deleted autoscaling schedule VqmMZqO ``` For the full flag set, see [`autoscaling-schedule delete`](/reference/cli/baseten/model-environment-autoscaling-schedule#delete). ## Manage schedules from the API The [Update environment](/reference/management-api/environments/update-an-environments-settings) endpoint manages the same schedule collection. Each entry in `schedules` is a complete create or replacement: include a schedule's `id` to replace it, or omit `id` to create a new one. Schedules you leave out of the list stay unchanged, and deletion stays explicit through `delete_schedules`. ```json PATCH /v1/models/{model_id}/environments/{env_name} theme={"system"} { "autoscaling_schedule_settings": { "timezone": "America/Los_Angeles", "schedules": [ { "cadence": "ONE_TIME", "name": "Launch day scale-up", "enabled": true, "start_at": "2026-09-01T16:00:00Z", "end_at": "2026-09-01T20:00:00Z", "autoscaling_settings": { "min_replica": 4, "max_replica": 16, "autoscaling_window": null, "scale_down_delay": null, "concurrency_target": null, "target_utilization_percentage": null, "target_in_flight_tokens": null, "max_scale_down_rate": null } } ] } } ``` The `cadence` field selects the timing shape: `DAILY` and `HOURLY` take the weekday and time-of-day fields, while `ONE_TIME` takes `start_at` and `end_at`. Every `autoscaling_settings` field is required; set a field to `null` to follow the environment's current value. One-time date-times must include a timezone offset, and Baseten converts them to UTC. The PATCH response reports the requested state, and schedule application reconciles asynchronously. Poll [Get environment](/reference/management-api/environments/get-an-environments-details) to confirm the applied settings. ## Next steps Match autoscaling settings to the shape of your traffic. Shrink the time between a scale-up and a served request. Understand the environments that schedules attach to. Apply autoscaling changes to a live deployment from a script. # Traffic patterns Source: https://docs.baseten.co/deployment/autoscaling/traffic-patterns Identify your traffic pattern and configure autoscaling settings to match. Different traffic patterns require different autoscaling configurations. Identify your pattern below for recommended starting settings. These are **starting points**, not final answers. Monitor your deployment's performance and adjust based on observed behavior. See [Autoscaling](/deployment/autoscaling/overview) for parameter details. ## Identify your pattern Not sure which pattern you have? Check your metrics: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar, then select your model. 2. Choose the **Metrics** tab. 3. Look at **Inference volume** and **Replicas** over the past week. 4. Compare to the patterns below. | You see... | Your pattern is... | | ----------------------------------------------------- | ------------------------------- | | Frequent small spikes that quickly return to baseline | [Jittery](#jittery-traffic) | | Sharp jumps that stay high for a while | [Bursty](#bursty-traffic) | | Long flat periods with occasional large bursts | [Scheduled](#scheduled-traffic) | | Gradual rises and falls, smooth curves | [Steady](#steady-traffic) | If traffic combines a daily pattern with sharp bursts, tune for the bursts, then measure cost during steady periods. ## Jittery traffic Small, frequent spikes that quickly return to baseline. ### Characteristics * Traffic repeatedly rises to at least twice its baseline, then quickly returns. * Spikes are short-lived and return to baseline quickly. * Often not real load growth, just temporary surges causing overreaction. ### Common causes * Consumer products with intermittent usage bursts. * Traffic splitting or A/B testing with low percentages. * Polling clients with synchronized intervals. ### Recommended settings | Parameter | Value | Why | | ------------------ | ----------------- | ----------------------------------------------- | | Autoscaling window | **2-5 minutes** | Smooth out noise, avoid reacting to every spike | | Scale-down delay | **300-600s** | Moderate stability | | Target utilization | **70%** | Default is fine | | Concurrency target | Benchmarked value | Start conservative | A longer autoscaling window averages out the jitter so the autoscaler doesn't react to every small spike. This trades reaction speed for stability when the spikes aren't sustained load increases. If you're still seeing oscillation with these settings, increase the scale-down delay before lowering target utilization. ## Bursty traffic ### Characteristics * Traffic **jumps sharply** (2x+ within 60 seconds). * Stays high for a sustained period before dropping. * Queuing and latency increase while replicas start. ### Common causes * Daily morning ramp-up (users starting their day). * Marketing events, product launches, viral moments. * Top-of-hour scheduled jobs or cron-triggered traffic. ### Recommended settings | Parameter | Value | Why | | ------------------ | ---------- | --------------------------------------------- | | Autoscaling window | **30-60s** | React quickly to genuine load increases | | Scale-down delay | **900s+** | Handle back-to-back waves without thrashing | | Target utilization | **50-60%** | More headroom absorbs the burst while scaling | | Min replicas | **≥2** | Redundancy + reduces cold start impact | A shorter window lets the autoscaler react sooner. A longer delay prevents scale-down between waves. Lower target utilization leaves capacity available while new replicas start. ### Pre-warming for predictable bursts To pre-warm before a predictable burst (morning ramp, scheduled events): 1. Before the expected spike, bump min replicas: ```bash Request theme={"system"} curl -X PATCH \ https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/autoscaling_settings \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{"min_replica": 5}' ``` 2. After the burst subsides, reset to your normal minimum: ```bash Request theme={"system"} curl -X PATCH \ https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/autoscaling_settings \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{"min_replica": 2}' ``` For peaks at known times, use an [autoscaling schedule](/deployment/autoscaling/schedules) to raise minimum replicas before traffic arrives. Start the schedule early enough to cover your deployment's observed cold start time. ## Scheduled traffic ### Characteristics * **Long periods of low or zero traffic**. * Large bursts tied to job schedules (hourly, daily, weekly). * Traffic patterns are predictable but infrequent. ### Common causes * ETL pipelines and data processing jobs. * Embedding backfills and batch inference. * Periodic evaluation or testing jobs. * Document processing triggered by user uploads. ### Recommended settings | Parameter | Value | Why | | ------------------ | --------------------------------------------------------------- | ----------------------------------------- | | Min replicas | **0** (if cold starts acceptable) or **1** (during job windows) | Cost savings when idle | | Scale-down delay | **Moderate to high** | Jobs often come in waves | | Autoscaling window | **60-120s** | Don't overreact to the first few requests | | Target utilization | **70%** | Default is fine | Scale-to-zero avoids replica charges during idle periods. The moderate window filters short-lived changes at the start of a batch. If jobs come in waves, a longer delay keeps replicas warm between them. ### Scheduled pre-warming [Autoscaling schedules](/deployment/autoscaling/schedules) pre-warm replicas without an external job or API key. For a batch job that starts at the top of each hour, create an hourly window that starts at `:55` and ends after the job completes. Set the schedule's `min_replica` to the warm capacity the job needs. Outside the window, the environment returns to its default settings. Start the window earlier if replicas take more than five minutes to become ready. Use your worst observed cold start as the initial lead time, then adjust it from deployment metrics. If an existing orchestration system must own the schedule, run the same updates through the Management API. This example raises the minimum five minutes before an hourly job: ```bash Terminal theme={"system"} 55 * * * * curl -X PATCH \ https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/autoscaling_settings \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{"min_replica": 3}' ``` This update restores scale-to-zero 30 minutes after the hour: ```bash Terminal theme={"system"} 30 * * * * curl -X PATCH \ https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/autoscaling_settings \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{"min_replica": 0}' ``` If the deployment has reached zero replicas when a batch starts, Baseten must [cold-start](/deployment/autoscaling/cold-starts) a replica before it can process the batch. Synchronous requests can wait while the replica starts. Setting `min_replica` to 0 allows the deployment to reach this state. For latency-sensitive batch jobs, set `min_replica` to at least 1 for the job window and start pre-warming before traffic arrives. ## Steady traffic ### Characteristics * Traffic **rises and falls gradually** over the day. * Classic diurnal pattern with no sharp edges. * Predictable, cyclical behavior. ### Common causes * Always-on inference APIs with consistent user base. * B2B applications with business-hours usage. * Production workloads with stable, mature traffic. ### Recommended settings | Parameter | Value | Why | | ------------------ | ------------ | ------------------------------------ | | Target utilization | **70-80%** | Uses more of each replica's capacity | | Autoscaling window | **60-120s** | Moderate reaction speed | | Scale-down delay | **300-600s** | Moderate | | Min replicas | **≥2** | Redundancy for production | Without sudden spikes, you don't need as much headroom. You can run replicas at higher utilization to reduce cost because load changes are gradual and predictable. The autoscaler has time to react. Start with the defaults. Monitor for a week, then gradually raise target utilization while watching p95 latency. ## Next steps * [Autoscaling](/deployment/autoscaling/overview): Full parameter documentation. * [Scale a deployment](/deployment/manage/scaling): Apply these settings from the dashboard or Management API. * [Troubleshooting autoscaling](/troubleshooting/deployments#autoscaling-issues): Diagnose and fix common problems. * [Truss configuration reference](/reference/truss-configuration): Configure `predict_concurrency` in your model. # CI/CD Source: https://docs.baseten.co/deployment/ci-cd Automate Truss deployments with GitHub Actions. Use the [Truss Push GitHub Action](https://github.com/marketplace/actions/truss-push) to deploy and validate Truss models or [Chains](/development/chain/deploy) from GitHub. Run it on pull requests to test a temporary deployment, or on merges to release a model to an environment. ## What happens during a run Depending on the deployment type and inputs, a run can include these steps: 1. **Load config**: For models, reads `config.yaml` from the Truss directory and extracts `model_metadata.example_model_input` for the predict step. The `predict-payload` input overrides that value. For Chains, detects the entrypoint class from the `.py` file. 2. **Deploy**: Pushes the model or Chain to Baseten. For a model, the action streams build logs into the GitHub Actions output. For a Chain, it prints deployment status changes and chainlet log URLs. For models, the action also names each deployment from git context: `PR-42_abc1234` for pull requests or `abc1234` for direct pushes. Override the generated name with `deployment-name`. 3. **Predict**: If the action resolves a payload that it doesn't skip, sends a predict request and reports latency. For streaming models when the payload includes `"stream": true`, it reports time-to-first-byte, token count, and tokens per second. 4. **Cleanup**: If `cleanup: true`, deactivates the newly created deployment. Set `cleanup: false` when a model workflow deploys to an environment or when you want to inspect a deployment manually. The job summary includes the time spent waiting for the deployment to become active after the push call returns. If prediction runs, the summary also includes predict metrics. A model run includes a direct link to its deployment logs. For a Chain, use the chainlet log URLs in the deploy output. ## Prerequisites Store your Baseten API key as an [encrypted secret](https://docs.github.com/en/actions/security-guides/encrypted-secrets) named `BASETEN_API_KEY` in your repository or organization settings. See [API keys](/organization/api-keys) for how to generate one. ## Deploy with the Baseten CLI The GitHub Action wraps deploy, predict validation, and cleanup in one step. If you only need to deploy, run the Baseten CLI directly in your workflow. The CLI reads your API key from the `BASETEN_API_KEY` environment variable, and `--wait` blocks the step until the deployment is active so a failed deploy fails the job: ```yaml .github/workflows/deploy.yml theme={"system"} name: Deploy to production on: push: branches: [main] jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Install the Baseten CLI run: brew tap basetenlabs/baseten && brew install baseten - name: Push the model env: BASETEN_API_KEY: ${{ secrets.BASETEN_API_KEY }} run: baseten model push --dir ./my-model --environment production --wait ``` For the full push surface, see [`model push`](/reference/cli/baseten/model#push). To add predict validation and cleanup to a CLI-based flow, see [Validate on pull request](#validate-on-pull-request) for the Action-based equivalent. ## Deploy to an environment on merge Deploy a validated model to a specific environment every time code merges to `main`. Create `.github/workflows/deploy.yml` and add the following: ```yaml .github/workflows/deploy.yml theme={"system"} name: Deploy to production on: push: branches: [main] jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: basetenlabs/action-truss-push@v0.1.2 with: truss-directory: "./my-model" baseten-api-key: ${{ secrets.BASETEN_API_KEY }} environment: "production" cleanup: false ``` Setting `environment` publishes the model deployment to the specified environment. Setting `cleanup: false` keeps the deployment active so it can serve traffic. For a model deployed to a [regional environment](/deployment/regional-environments), add `regional-environment: true` to the action's `with` block. The action then sends its predict validation request through the regional endpoint. ## Validate on pull request Catch model regressions before they reach production. The action deploys, runs a predict request, and tears down the deployment inside the PR check. Create `.github/workflows/validate-model.yml` and add the following: ```yaml .github/workflows/validate-model.yml theme={"system"} name: Validate model on: pull_request: branches: [main] jobs: validate: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: basetenlabs/action-truss-push@v0.1.2 with: truss-directory: "./my-model" baseten-api-key: ${{ secrets.BASETEN_API_KEY }} ``` The action reads `model_metadata.example_model_input` from your `config.yaml` to build the predict request. With the default (`cleanup: true`), the deployment is deactivated after validation, so no resources are left running. ## Deploy a Chain Deploy a Baseten Chain from a Python source file. The action auto-detects Chains when `truss-directory` points to a `.py` file: ```yaml theme={"system"} - uses: basetenlabs/action-truss-push@v0.1.2 with: truss-directory: "./chains/my_chain.py" baseten-api-key: ${{ secrets.BASETEN_API_KEY }} model-name: "my-rag-chain" cleanup: false predict-payload: '{"query": "What is Baseten?"}' ``` For Chains, provide the predict payload explicitly with `predict-payload` because there's no `config.yaml` to read example input from. In `v0.1.2`, the environment inputs don't promote a Chain. Leave `regional-environment` set to `false` to validate the new Chain deployment through its deployment-specific endpoint. See the [`regional-environment` input](/reference/ci/github-action#param-regional-environment) for regional validation behavior. ## Deploy multiple models Use a matrix strategy to deploy each model in your repository as a separate job. Create `.github/workflows/deploy-all.yml` and add the following: ```yaml .github/workflows/deploy-all.yml theme={"system"} name: Deploy models on: push: branches: [main] jobs: deploy: runs-on: ubuntu-latest strategy: fail-fast: false matrix: model: - path: models/text-classifier - path: models/image-generator - path: models/embeddings steps: - uses: actions/checkout@v4 - uses: basetenlabs/action-truss-push@v0.1.2 with: truss-directory: ${{ matrix.model.path }} baseten-api-key: ${{ secrets.BASETEN_API_KEY }} environment: "production" cleanup: false ``` Each matrix entry runs as a separate job. If one model fails, the others still deploy. ## Custom predict validation Override the default predict payload when your model needs a specific input shape that differs from `model_metadata.example_model_input`: ```yaml theme={"system"} - uses: basetenlabs/action-truss-push@v0.1.2 with: truss-directory: "./my-model" baseten-api-key: ${{ secrets.BASETEN_API_KEY }} predict-payload: '{"prompt": "Hello, world!", "max_new_tokens": 128}' predict-timeout: 60 ``` The action skips the predict step when the resolved JSON value is missing, `null`, `false`, `0`, an empty string (`""`), an empty object (`{}`), or an empty array (`[]`). A skipped predict step doesn't validate the deployment. ## Deploy with labels Attach metadata labels to track model deployments in your CI pipeline: ```yaml theme={"system"} - uses: basetenlabs/action-truss-push@v0.1.2 with: truss-directory: "./my-model" baseten-api-key: ${{ secrets.BASETEN_API_KEY }} labels: '{"team": "ml-platform", "triggered-by": "ci"}' ``` For more information, see [Label deployments](/deployment/manage/labels). ## Override model name Set a custom model name instead of using the name from `config.yaml`: ```yaml theme={"system"} - uses: basetenlabs/action-truss-push@v0.1.2 with: truss-directory: "./my-model" baseten-api-key: ${{ secrets.BASETEN_API_KEY }} model-name: "my-custom-name" ``` ## Use action outputs The action exposes outputs you can reference in downstream steps. This example posts the deploy time as a PR comment: ```yaml theme={"system"} steps: - uses: actions/checkout@v4 - uses: basetenlabs/action-truss-push@v0.1.2 id: deploy with: truss-directory: "./my-model" baseten-api-key: ${{ secrets.BASETEN_API_KEY }} - name: Comment on PR if: github.event_name == 'pull_request' uses: actions/github-script@v7 with: script: | github.rest.issues.createComment({ owner: context.repo.owner, repo: context.repo.repo, issue_number: context.issue.number, body: `Post-push wait to active: ${{ steps.deploy.outputs.deploy-time-seconds }} seconds. Status: ${{ steps.deploy.outputs.status }}` }) ``` See the full list of inputs and outputs in the [Truss Push GitHub Action reference](/reference/ci/github-action). ## Troubleshooting ### `deploy_timeout` The default timeout is 45 minutes, which accommodates large builds like TRT-LLM. For smaller models, reduce `deploy-timeout-minutes` to fail faster. If your model needs more time, increase the value. ### `deploy_failed` Verify the `BASETEN_API_KEY` secret first. For a model, check `config.yaml` and expand the build output in the GitHub Actions UI. For a Chain, check the source file and follow the chainlet log URLs in the deploy output. ### `predict_failed` Verify the predict payload shape matches what your model expects. Check `model_metadata.example_model_input` in `config.yaml`, or override it with `predict-payload`. For Chains, provide the predict payload explicitly. ### `cleanup_failed` The deployment may still be running. Deactivate it manually from the [Baseten dashboard](https://app.baseten.co). ### 429 too many requests The action calls Management API endpoints that are rate limited per API key. Matrix jobs that fan out across many models can exceed the per-endpoint limits. See [Management API rate limits](/reference/management-api/rate-limits) for thresholds and backoff guidance. ### No predict output The action skips prediction when the resolved JSON value is missing, `null`, `false`, `0`, an empty string (`""`), an empty object (`{}`), or an empty array (`[]`). For a model, add an example input to `config.yaml` that isn't one of these values. For a Chain, set `predict-payload` to a value that the action doesn't skip. ### `Team selection required but running in a non-interactive context` For a model workflow, your API key has access to multiple teams and Truss can't infer a single target team without a prompt. Pass the team explicitly with the `team` input on the action. If you invoke `truss push` directly, use `--team `. ```yaml theme={"system"} - uses: basetenlabs/action-truss-push@v0.1.2 with: truss-directory: "./my-model" baseten-api-key: ${{ secrets.BASETEN_API_KEY }} team: "ml-platform" ``` See [Deploy to a team](/organization/teams#use-the-truss-cli) for the team-resolution rules. Requires Truss `0.18.3` or later. The `team` input in `action-truss-push@v0.1.2` applies to models only. For a Chain workflow, use an API key that resolves to the intended team without an interactive selection. ## Next steps Promote validated deployments to production without downtime. Manage staging and production environments for your models. Scale, promote, and clean up deployments from the Baseten CLI or Management API. # Concepts Source: https://docs.baseten.co/deployment/concepts Deployments, environments, resources, autoscaling, and CI/CD on Baseten. A [deployment](/deployment/deployments) is a version of your model running on Baseten with its own API endpoint. This page explains how deployments, environments, resources, autoscaling, and CI/CD fit together. To create, promote, scale, or deactivate a deployment, see [Manage deployments](/deployment/manage/overview). ## Deployments A deployment runs one version of your model on a selected instance type. Each `baseten model push` creates a deployment. Multiple deployments of the same model can run at once, allowing you to test a new version without changing production traffic. Deactivate a deployment to stop serving and billing, or delete it when you no longer need it. For rapid iteration, use `baseten model push --watch` to create a **development deployment**, a mutable instance that live-reloads as you edit your model code. You can promote a development deployment directly to an environment, which creates a published deployment. Baseten dashboard showing multiple model deployments ### Model lab deployments For deployment restrictions on models distributed by model labs, see [Customer availability and artifact access](/labs/platform#customer-availability-and-artifact-access). ## Environments [Environments](/deployment/environments) provide stable endpoints that persist as you release new deployments. A typical setup uses one environment for testing and another for production traffic. Each environment has its own endpoint, autoscaling settings, and metrics. Promoting a deployment moves the environment's traffic to that version without changing the URL called by your application. Deployment environments with development and production endpoints ## Resources Every deployment runs on an [instance type](/deployment/resources) that defines its GPU, CPU, and memory. Set the instance type in `config.yaml`, or change it for a published deployment in the Baseten dashboard. Select an instance based on the model's memory requirements, latency target, and expected traffic. Resource configuration showing GPU instance type selection ## Autoscaling [Autoscaling](/deployment/autoscaling/overview) adjusts the number of replicas as request load changes. Configure the minimum and maximum replicas, concurrency target, and scale-down delay. A deployment can scale to zero when idle, so a synchronous request can wait for a replica to start. See [Cold starts](/deployment/autoscaling/cold-starts) for ways to reduce startup time. Autoscaling configuration with replica count and concurrency settings For the mechanics of how the autoscaler tracks in-flight requests and adjusts replicas, see [How Baseten works](/concepts/howbasetenworks#autoscaling). For engine-specific autoscaling settings (BEI and Engine-Builder-LLM), see [Autoscaling engines](/engines/performance-concepts/autoscaling-engines). ## Request lifecycle Before your model code runs, each request passes through authentication, routing, and replica selection. See [Request lifecycle](/deployment/autoscaling/request-lifecycle) to understand queuing, load shedding, timeouts, and HTTP status codes. ## CI/CD Use the [Truss Push GitHub Action](/deployment/ci-cd) to deploy a model from a Git repository, validate it with a predict request, and optionally promote it to production. Configure the workflow to run for selected branches, pushes, or pull requests. # Deployments Source: https://docs.baseten.co/deployment/deployments Understand published and development deployments, how they relate to environments, and which lifecycle action to use. A deployment is one version of a model running on Baseten. It has its own ID, model artifact, instance type, autoscaling settings, replicas, and API endpoint. A model can have multiple deployments at the same time. Use this page to understand the deployment resource and its relationship to environments. To perform an operation, see [Manage deployments](/deployment/manage/overview). To send an inference request, see [Call a model](/inference/calling-your-model). ## Deployment types ### Published deployments By default, `baseten model push` creates a published deployment. Its model artifact doesn't live reload, so a code or configuration change requires another push. Published model deployments support the full set of autoscaling and environment promotion settings. A published deployment can serve requests through its deployment-specific endpoint without belonging to an environment. ### Development deployment Use `baseten model push --watch` for a model or `baseten truss chains push --watch` for a Chain to create a development deployment. Development deployments support a fast edit-and-reload loop: * Source changes live reload without another deployment. * The deployment uses one replica when active and scales to zero when idle. * The deployment can't scale beyond one replica or use zero-downtime updates. Promoting a development deployment to an environment creates a new published deployment with its own ID and starts an image build. After the deployment becomes ready, Baseten runs the environment's configured promotion workflow. The promotion completes when the environment points to the new published deployment. ## Environments and promotion An [environment](/deployment/environments) gives a model a stable endpoint and a reusable set of resource, autoscaling, and promotion settings. The environment points to the deployment that currently serves its traffic. * A deployment doesn't need an environment to serve requests. * The production environment exists by default. You can add environments such as staging. * Promoting a deployment changes which version serves the environment endpoint without changing that endpoint's URL. * Promotion can reuse the source deployment or create a new published deployment, depending on the source type and the environment settings. ### Rolling deployments Rolling deployment is an environment promotion strategy, not a deployment type. It incrementally provisions the candidate deployment, shifts traffic, and removes the previous deployment's replicas. You can pause, resume, cancel, or force-complete the promotion. For more information, see [Rolling deployments](/deployment/rolling-deployments). ### Canary deployments (deprecated) Baseten has deprecated canary deployments. Use [rolling deployments](/deployment/rolling-deployments) for incremental traffic shifting. ## Deployment metadata Baseten addresses a deployment by its model ID and deployment ID. Names and labels help people and automation identify deployments. A deployment name is a human-readable identifier and doesn't change the deployment's API path. Set it with [`truss push --deployment-name`](/reference/cli/truss/push), or rename it later with the Baseten CLI or in the Baseten dashboard: ```bash Command theme={"system"} baseten model deployment rename --model-id --deployment-id --new-name ``` ```txt Output theme={"system"} Renamed deployment to ``` For more information, see [`deployment rename`](/reference/cli/baseten/model-deployment#rename). [Deployment labels](/deployment/manage/labels) are JSON key-value metadata for ownership, source control, or CI information. You can't change a published deployment's labels after creation. A later development push that includes labels replaces the development deployment's labels. A push that omits labels preserves them. ## Deployment lifecycle Use [Manage deployments](/deployment/manage/overview) to promote, scale, inspect, deactivate, activate, or delete a deployment. The task guides explain each operation and its effect on inference traffic. ## Next steps Scale, promote, inspect, deactivate, and delete deployments. Configure stable endpoints and promotion behavior. Configure replica capacity and scaling behavior. Send inference requests to a deployment or environment. # Environments Source: https://docs.baseten.co/deployment/environments Manage your model's release cycles with environments. An environment is a named release target for a model, such as production or staging. It has a stable endpoint and its own resource, autoscaling, monitoring, and promotion settings. Promote a new deployment to update the version that serves the environment without changing the endpoint your client calls. Baseten dashboard showing model environments and their deployments Use a separate environment to validate a deployment without changing production traffic. ## Deployment management Use environments to validate and release deployments with: * Automated tests and evaluations. * Manual testing in pre-production. * Gradual traffic shifts with [rolling deployments](/deployment/rolling-deployments). When you promote a deployment, Baseten applies the environment's scaling and monitoring settings: * Dedicated API endpoint. For more information, see [Predict endpoints](/reference/inference-api/overview#predict-endpoints). * Autoscaling controls: Scale behavior is managed per environment. * Monitoring and metrics: scope [logs](/observability/logs#scope-by-environment-or-deployment) and [metrics](/observability/metrics) to the environment in the dashboard, or [export environment metrics](/observability/export-metrics/overview) to your own observability stack. The production environment operates like any other environment but has restrictions: * It can't be deleted unless the entire model is removed. * You can't create additional environments named "production." ## Manage environments from the Baseten CLI The [`baseten model environment`](/reference/cli/baseten/model-environment) commands work on the same environments as the dashboard and the Management API: * `baseten model environment list --model-id `: list a model's environments. * `baseten model environment describe --model-id --environment production`: inspect an environment's current deployment, autoscaling, and promotion settings. * `baseten model environment update-autoscaling` and `update-promotion`: change an environment's settings; see [schedule autoscaling](/deployment/autoscaling/schedules) and [rolling deployments](/deployment/rolling-deployments). * `baseten model deployment promote --model-id --deployment-id `: promote a deployment into an environment; see [Manage deployments](/deployment/manage/overview). ## Custom environments Each workspace has a configured per-model environment limit. In addition to the standard production environment, you can create custom environments until the model reaches that limit. Create them from the model management page in the Baseten dashboard or through the [create environment endpoint](/reference/management-api/environments/create-an-environment) in the Management API. ## Deployment promotion When you promote a deployment to an environment, Baseten associates the deployment with that environment and applies the environment's autoscaling settings. If Baseten can reuse the deployment directly, promotion doesn't create a new deployment ID. Otherwise, Baseten creates a deployment with a unique ID, initializes its resources, and replaces the existing deployment in that environment. A new deployment is created when: * You promote a development deployment. * The deployment is already associated with another environment. * The environment has a different instance type or resource profile. * [Re-deploy on promotion](#re-deploy-on-promotion) is enabled. When Baseten creates a new deployment, it applies the environment's autoscaling settings to that deployment. After promotion completes, the environment's [promotion cleanup strategy](/deployment/rolling-deployments#deployment-cleanup) controls the deployment it replaced. If Baseten creates a new deployment from a published source, the new deployment reuses the source image. Promoting a development deployment creates a new published deployment from its current state and triggers an image build. ### Published deployment promotion If Baseten can reuse a published deployment, it updates that deployment's autoscaling settings to match the environment. If promotion creates a new deployment, the source published deployment keeps its existing settings. ## Direct deployment to an environment You can deploy directly to a named environment by passing `--environment` to `baseten model push`: ```sh Terminal theme={"system"} cd my_model/ baseten model push --environment {environment_name} ``` The Truss CLI equivalent is `truss push --environment {environment_name}`. Only one active promotion per environment is allowed at a time. ## Environment access in code The environment name is available in `model.py` through the `environment` keyword argument: ```python model/model.py theme={"system"} def __init__(self, **kwargs): self._environment = kwargs["environment"] ``` You can use the environment in your `load()` method to configure per-environment behavior: ```python model/model.py theme={"system"} def load(self): if self._environment.get("name") == "production": self.setup_sentry() self.model = self.load_production_weights() else: self.model = self.load_default_weights() ``` If you use environment-specific configuration in `load()`, you'll need to enable re-deploy on promotion to ensure the environment is correctly initialized after each promotion. See [Re-deploy on promotion](#re-deploy-on-promotion) for details. The `environment` keyword argument is only available to Python Truss models. Custom servers read the environment name from the filesystem instead. See [Environment name](/development/model/custom-server#environment-name). ## Re-deploy on promotion By default, promoting a deployment reuses the existing deployment when possible. This is the fastest promotion path, but it means `load()` doesn't re-run. Any environment-specific configuration set during the original `load()` call persists, even if the deployment moves to a different environment. You can configure an environment to create a fresh deployment every time you promote to it. For a published source deployment, the new deployment reuses the source image and reruns `load()` with the target environment's context, so environment-specific configuration takes effect. Enable this if your `load()` method uses `kwargs["environment"]` to configure per-environment behavior, or if you promote the same source deployment to multiple environments and want each to get a fresh deployment. To enable re-deploy on promotion: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar, then select your model. 2. Open the environment's settings. 3. Toggle **Re-deploy when promoting**. You can also set it through the [update environment settings endpoint](/reference/management-api/environments/update-an-environments-settings). If you promote a deployment that's already associated with an environment, Baseten creates a new deployment regardless of this setting. ## Regional environments Regional environments constrain replicas to a designated geographic region and provide a dedicated endpoint for region-specific inference routing. Baseten configures the regional restriction for an environment name within a team. See [Regional environments](/deployment/regional-environments) for setup, required-endpoint behavior, endpoint formats, and request restrictions. ## Environment deletion You can delete environments, except for production. To remove a production deployment, first promote another deployment to production or delete the entire model. * Deleted environments are removed from the overview but remain in billing history. * They don't consume resources after deletion. * API requests to a deleted environment return a 404 error. # Label deployments Source: https://docs.baseten.co/deployment/manage/labels Add labels when you push a deployment, then view them in the dashboard, the Baseten CLI, or the Management API. Labels are a JSON object stored with a deployment. Use them to record metadata such as the owning team, source commit, or CI run. Published deployment labels remain fixed after creation. To change them, create another deployment with new labels. For a development deployment, passing labels on a later push replaces its current labels. Omitting labels preserves them. A label object can be up to 2 KB after JSON serialization. Label keys and string values can contain printable ASCII characters except spaces. ## Set labels **To label a deployment**: ```bash theme={"system"} baseten model push --labels '{"team": "ml-platform", "env": "staging"}' ``` For more information, see [`baseten model push`](/reference/cli/baseten/model). **To label a deployment**: ```bash theme={"system"} truss push --labels '{"team": "ml-platform", "env": "staging"}' ``` For more information, see [`truss push`](/reference/cli/truss/push). **To label a deployment**: ```python theme={"system"} import truss truss.push("./my-model", labels={"team": "ml-platform", "env": "staging"}) ``` For more information, see [`truss.push()`](/reference/sdk/truss/push). **To label a deployment**: Add `labels` to the `deployment` object in both the prepare and create requests: ```json theme={"system"} { "labels": { "team": "ml-platform", "env": "staging" } } ``` For the complete prepare, upload, and create sequence, see [Create a model with the REST API](/examples/create-a-model-with-rest). **To label a deployment**: ```yaml theme={"system"} - uses: basetenlabs/action-truss-push@v0.1.2 with: truss-directory: "./my-model" baseten-api-key: ${{ secrets.BASETEN_API_KEY }} labels: '{"team": "ml-platform", "triggered-by": "ci"}' ``` For more information, see the [deploy GitHub Action](/reference/ci/github-action) and [Deploy with labels](/deployment/ci-cd#deploy-with-labels). ## View labels After you push the deployment, view its labels in the dashboard, the Baseten CLI, or the Management API: **To view a deployment's labels**: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar. 2. Select your model. 3. Select the deployment under **Deployments**. 4. Find **Labels** in the deployment details, below **Truss config**. **To view a deployment's labels**: JSON output includes labels. The text summary omits them. Print the whole deployment record with `--output json`: ```bash Command theme={"system"} baseten model deployment describe --model-id abc123 --deployment-id def456 --output json ``` ```json Output theme={"system"} { "id": "def456", "name": "deployment-1", "model_id": "abc123", "status": "ACTIVE", "instance_type_name": "1x2 - 1 vCPU, 2 GiB RAM", "labels": { "env": "staging", "team": "ml-platform" }, ... } ``` Print just the labels with `--jq`: ```bash Command theme={"system"} baseten model deployment describe --model-id abc123 --deployment-id def456 --jq '.labels' ``` ```json Output theme={"system"} { "env": "staging", "team": "ml-platform" } ``` For more information, see [`baseten model deployment describe`](/reference/cli/baseten/model-deployment). **To view a deployment's labels**: ```bash Request theme={"system"} curl "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}" \ -H "Authorization: Bearer $BASETEN_API_KEY" ``` ```json Response theme={"system"} { "id": "def456", "name": "deployment-1", "model_id": "abc123", "status": "ACTIVE", "labels": { "team": "ml-platform", "env": "staging" }, ... } ``` For more information, see [Get a model's deployment by ID](/reference/management-api/deployments/gets-a-models-deployment-by-id). ## Next steps Use labels in CI/CD to compare the deployment running on Baseten with the commit or workflow run that created it. * [Manage the deployment lifecycle](/deployment/manage/lifecycle) to promote or retire a labeled deployment. * [CI/CD](/deployment/ci-cd) to attach labels automatically from GitHub Actions. * [Deployments](/deployment/deployments) for how deployments, environments, and promotion fit together. # Manage the deployment lifecycle Source: https://docs.baseten.co/deployment/manage/lifecycle Promote, deactivate, activate, and delete deployments from the dashboard, Baseten CLI, or Management API. Manage your deployment through its lifecycle. After you push a model, promote its deployment to serve an environment, deactivate it to stop compute spend, activate it to bring it back, and delete it when you no longer need it. For what each deployment state means, see [Deployments](/deployment/deployments). ## Promote to an environment Promote a validated deployment to production (or a custom environment) to route that environment's traffic to it. What happens to the previously promoted deployment is controlled by the target environment's promotion cleanup strategy; see [Environments](/deployment/environments) for the concepts and [Rolling deployments](/deployment/rolling-deployments) for incremental traffic shifting. Promotion can create a new published deployment with a different ID. For example, Baseten creates one when you promote a development deployment. A promotion can also reuse the source deployment. In both cases, use the `id` in the promotion response as the promoted deployment ID. The request can return while that deployment is still `BUILDING`, `DEPLOYING`, or `UPDATING`, before the target environment points to it. The cURL examples use [`jq`](https://jqlang.github.io/jq/) to extract fields from JSON responses. **To promote a deployment**: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar. 2. Select your model. 3. Select the deployment under **Deployments**. 4. Choose **Promote to...**. 5. In the **Promote deployment** dialog, choose the target environment. 6. Choose **Promote** to confirm. **To promote a deployment**: Promote to production, the CLI's default target, and save the returned deployment ID: ```bash Command theme={"system"} target_environment=production promoted_deployment_id="$( baseten model deployment promote \ --model-id \ --deployment-id \ --yes \ --jq '.id' )" printf 'Promoted deployment ID: %s\n' "$promoted_deployment_id" ``` ```txt Output theme={"system"} Promoted deployment ID: ``` Promote to another environment with `--environment ` and save the returned deployment ID: ```bash Command theme={"system"} target_environment=staging promoted_deployment_id="$( baseten model deployment promote \ --model-id \ --deployment-id \ --environment "$target_environment" \ --yes \ --jq '.id' )" printf 'Promoted deployment ID: %s\n' "$promoted_deployment_id" ``` ```txt Output theme={"system"} Promoted deployment ID: ``` **To promote a deployment**: The response examples below show development deployment promotions. The returned published deployment is still `BUILDING`, so the target environment doesn't point to it yet. A response for a reused published deployment can show `ACTIVE` or `UPDATING` and include the target environment's name. Promote to production and save the returned deployment ID: ```bash Request theme={"system"} target_environment=production promotion_response="$( curl -sS -X POST "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/promote" \ -H "Authorization: Bearer $BASETEN_API_KEY" )" promoted_deployment_id="$(printf '%s' "$promotion_response" | jq -r '.id')" printf '%s\n' "$promotion_response" | jq '{id, is_production, is_development, status, environment}' ``` ```json Response theme={"system"} { "id": "", "is_production": false, "is_development": false, "status": "BUILDING", "environment": null } ``` Promote to another environment by sending the source deployment ID to that environment's promote endpoint. Save the ID from the response: ```bash Request theme={"system"} target_environment=staging promotion_response="$( curl -sS -X POST "https://api.baseten.co/v1/models/{model_id}/environments/$target_environment/promote" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -H "Content-Type: application/json" \ -d '{"deployment_id": "{deployment_id}"}' )" promoted_deployment_id="$(printf '%s' "$promotion_response" | jq -r '.id')" printf '%s\n' "$promotion_response" | jq '{id, is_production, is_development, status, environment}' ``` ```json Response theme={"system"} { "id": "", "is_production": false, "is_development": false, "status": "BUILDING", "environment": null } ``` **To promote from CI/CD**: Each push to `main` deploys the Truss and promotes the new deployment to production: ```yaml .github/workflows/deploy.yml theme={"system"} name: Deploy to production on: push: branches: [main] jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: basetenlabs/action-truss-push@v0.1.2 with: truss-directory: "./my-model" baseten-api-key: ${{ secrets.BASETEN_API_KEY }} environment: "production" cleanup: false ``` For more information, see [CI/CD](/deployment/ci-cd) and the [Truss Push GitHub Action reference](/reference/ci/github-action). The promotion request can return before the target environment changes. The promotion is complete when the target environment's `current_deployment.id` matches the returned `promoted_deployment_id`. Check the environment, rather than the source deployment: **To verify the promotion**: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar. 2. Select your model. 3. On the model's overview, confirm the environment card's **Current deployment** shows the promoted deployment. **To verify the promotion**: Run this check again while the promotion is in progress: ```bash Command theme={"system"} current_deployment_id="$( baseten model environment describe \ --model-id \ --environment "$target_environment" \ --jq '.current_deployment.id // empty' )" if [ "$current_deployment_id" = "$promoted_deployment_id" ]; then printf 'Promotion complete: %s\n' "$current_deployment_id" else printf 'Promotion not complete\n' fi ``` ```txt Output theme={"system"} Promotion complete: ``` **To verify the promotion**: Run this check again while the promotion is in progress: ```bash Request theme={"system"} environment_response="$( curl -sS "https://api.baseten.co/v1/models/{model_id}/environments/$target_environment" \ -H "Authorization: Bearer $BASETEN_API_KEY" )" current_deployment_id="$( printf '%s' "$environment_response" | jq -r '.current_deployment.id // empty' )" if [ "$current_deployment_id" = "$promoted_deployment_id" ]; then printf 'Promotion complete: %s\n' "$current_deployment_id" else printf 'Promotion not complete\n' fi ``` ```txt Output theme={"system"} Promotion complete: ``` For more information, see the [promote deployment](/reference/management-api/deployments/promote/promotes-a-deployment-to-production) and [promote to environment](/reference/management-api/deployments/promote/promotes-a-deployment-to-an-environment) endpoints. ## Deactivate a deployment Deactivate a deployment to stop compute spend without deleting it. The deployment keeps its configuration and stays visible in the dashboard, but releases its replicas. Requests to a deactivated deployment fail with a `400` error; see [troubleshooting](/troubleshooting/deployments#issue-requests-fail-with-model-version-is-deactivated) for the error and recovery. **To deactivate a deployment**: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar. 2. Select your model. 3. Select the deployment under **Deployments**. 4. Choose **Deactivate deployment**. 5. Choose **Yes, deactivate** to confirm. **To deactivate a deployment**: ```bash Command theme={"system"} baseten model deployment deactivate --model-id --deployment-id ``` ```txt Output theme={"system"} Deactivated deployment ``` The CLI prompts for confirmation; pass `--yes` to skip it (required when scripting). **To deactivate a deployment**: ```bash Request theme={"system"} curl -X POST "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/deactivate" \ -H "Authorization: Bearer $BASETEN_API_KEY" ``` ```json Response theme={"system"} {"success": true} ``` The deployment's `status` moves to `INACTIVE` within a few seconds. If you want the deployment to keep serving but stop paying for idle replicas, [scale to zero](/deployment/manage/scaling#scale-to-zero) instead. Deactivating also releases the deployment's slot against your workspace's [deployment limit](/troubleshooting/deployments#issue-youve-reached-the-maximum-number-of-deployed-models). Scaling to zero doesn't, since the endpoint stays live. ## Activate a deployment Activate an inactive deployment to bring it back. Activation redeploys the model, so the deployment passes through `DEPLOYING` before reaching `ACTIVE`. **To activate a deployment**: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar. 2. Select your model. 3. Select the inactive deployment under **Deployments**. 4. Choose **Activate deployment**. **To activate a deployment**: ```bash Command theme={"system"} baseten model deployment activate --model-id --deployment-id ``` ```txt Output theme={"system"} Activated deployment ``` **To activate a deployment**: ```bash Request theme={"system"} curl -X POST "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/activate" \ -H "Authorization: Bearer $BASETEN_API_KEY" ``` ```json Response theme={"system"} {"success": true} ``` For more information, see [`activate`](/reference/cli/baseten/model-deployment#activate) and the [activate endpoint](/reference/management-api/deployments/activate/activates-a-deployment). ## Delete a deployment Delete deployments to clean up finished experiments and stale versions. Deletion is irreversible, and requests to a deleted deployment return `404`. [Deactivate](#deactivate-a-deployment) instead if you might need the deployment again. A deployment that's associated with an environment, or is the only deployment of a model, can't be deleted; [push a new deployment](/reference/cli/truss/push) and promote it first, or delete the whole model. **To delete a deployment**: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar. 2. Select your model. 3. Select the deployment under **Deployments**. 4. Choose **Delete deployment** and confirm. The button is disabled while the deployment serves an environment or is the model's only deployment. **To delete a deployment**: ```bash Command theme={"system"} baseten model deployment delete --model-id --deployment-id ``` ```txt Output theme={"system"} Deleted deployment ``` **To delete a deployment**: ```bash Request theme={"system"} curl -X DELETE "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}" \ -H "Authorization: Bearer $BASETEN_API_KEY" ``` ```json Response theme={"system"} {"id": "{deployment_id}", "deleted": true, "model_id": "{model_id}"} ``` Deleting a model removes all of its deployments. See [`baseten model delete`](/reference/cli/baseten/model#delete) or the [delete model endpoint](/reference/management-api/models/deletes-a-model-by-id). For more information, see the Management API [delete deployment endpoint](/reference/management-api/deployments/deletes-a-models-deployment-by-id) and the Baseten CLI [`deployment delete`](/reference/cli/baseten/model-deployment#delete) reference. ## Next steps * [Scale a deployment](/deployment/manage/scaling) to change replica behavior without a lifecycle change. * [Pull logs and metrics](/deployment/manage/logs-and-metrics) to verify a deployment's health after a transition. * [CI/CD](/deployment/ci-cd) to run deploys and promotions from GitHub Actions. # Pull logs and metrics Source: https://docs.baseten.co/deployment/manage/logs-and-metrics Fetch and stream logs and metrics for a deployment or an environment from the Baseten CLI or Management API for debugging and scripting. When a deployment misbehaves, start with its logs and metrics. This page covers reading both from the dashboard, the CLI, and the Management API. For metric definitions and Prometheus-style export, see [Metrics](/observability/metrics) and the other Observability pages. ## Fetch and stream logs Pull logs to debug an incident, filter for an error, or use the output in a script. Fetch a time window up to 7 days back: **To view logs**: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar. 2. Select your model. 3. Select the deployment under **Deployments**. 4. Choose **Logs**. Filter by level or replica, search, or follow live output with the controls at the top of the logs view. **To fetch logs**: ```bash Command theme={"system"} baseten model deployment logs --model-id --deployment-id --since 1h ``` ```txt Output theme={"system"} [2026-07-07 10:33:58]: Deploy was a success. [2026-07-07 10:33:44]: (nvsmp) Completed model.load() execution in 6 ms [2026-07-07 10:33:44]: (nvsmp) Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit) ``` The parenthesized prefix on each line is the replica that emitted it. **To stream live logs**: Pass `--tail` instead of a time window: ```bash Command theme={"system"} baseten model deployment logs --model-id --deployment-id --tail ``` ```txt Output theme={"system"} [2026-07-07 10:33:44]: (nvsmp) Completed model.load() execution in 6 ms [2026-07-07 10:33:58]: Deploy was a success. ... ``` Streaming continues until the deployment leaves a runnable state or you interrupt with Ctrl-C. For machine-readable output, add `--output jsonl` to stream one JSON log entry per line: ```bash Command theme={"system"} baseten model deployment logs --model-id --deployment-id --tail --output jsonl ``` ```json Output theme={"system"} {"level":"INFO","message":"Completed model.load() execution in 6 ms","replica":"nvsmp","timestamp":"1783445624380306972"} {"level":"INFO","message":"Deploy was a success.","replica":"","timestamp":"1783445638726940011"} ... ``` Narrow the stream with `--jq`: `--jq '.message'` extracts one field per line, and `--jq 'select(.level=="ERROR") | .message'` keeps only error lines. **To fetch logs**: ```bash Request theme={"system"} curl "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/logs" \ -H "Authorization: Bearer $BASETEN_API_KEY" ``` ```json Response theme={"system"} { "logs": [ { "timestamp": "1783445638726940011", "message": "Deploy was a success.", "replica": "", "request_id": null, "level": "INFO" }, { "timestamp": "1783445624380306972", "message": "Completed model.load() execution in 6 ms", "replica": "nvsmp", "request_id": null, "level": "INFO" } ] } ``` The window defaults to the last 30 minutes; pass `start_epoch_millis` to widen it, up to 7 days. Timestamps are epoch nanoseconds. The API fetches fixed windows only; to follow logs live, use the CLI's `--tail`. For more information about filters like log level, search patterns, and request ID, see [`logs`](/reference/cli/baseten/model-deployment#logs) and the [logs endpoint](/reference/management-api/deployments/get-deployment-logs) (beta). ## Fetch metrics Pull metrics to check a deployment's health from a script: replica count, request volume, and end-to-end latency quantiles. **To view metrics**: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar. 2. Select your model. 3. Select the deployment under **Deployments**. 4. Choose **Metrics**. The metrics view charts request rates by status code, latency percentiles, and replica counts. Set the window with the time-range selector at the top. **To fetch metrics**: ```bash Command theme={"system"} baseten model deployment metrics --model-id --deployment-id ``` ```txt Output theme={"system"} METRIC QUANTILE STAT VALUE baseten_replicas_active 1 baseten_end_to_end_response_time_seconds 0.5 - baseten_end_to_end_response_time_seconds 0.9 - baseten_end_to_end_response_time_seconds 0.95 - baseten_end_to_end_response_time_seconds 0.99 - baseten_end_to_end_response_time_seconds avg - ``` Latency values show `-` until the deployment has served requests in the window; once it has, a STATUS column also breaks out request counts per response code. The default is a current snapshot; pass `--mode summary` or `--mode series` with `--since` to aggregate over a window instead. **To fetch metrics**: ```bash Request theme={"system"} curl "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/metrics" \ -H "Authorization: Bearer $BASETEN_API_KEY" ``` ```json Response theme={"system"} { "start_epoch_millis": 1783445679297, "end_epoch_millis": 1783445679297, "mode": "CURRENT", "step_seconds": null, "metric_descriptors": [ { "name": "baseten_replicas_active", "unit_hint": "COUNT", "kind": "GAUGE", "label_sets": [{}] } ], "metric_values": [ { "start_epoch_millis": 1783445679297, "values": [[1.0]] } ] } ``` `metric_values` lines up with `metric_descriptors` by position: the first value array belongs to the first metric, the second to the second, and so on. For more information about modes, windows, and metric selection, see [`metrics`](/reference/cli/baseten/model-deployment#metrics) and the [metrics endpoint](/reference/management-api/deployments/get-deployment-metrics) (beta). For what each metric means, see [Metrics](/observability/metrics). ## Fetch environment logs and metrics Deployment scope answers "what is this specific deployment doing?" Environment scope answers "what is production doing right now?", which is usually the more useful operational question. An [environment](/deployment/environments) spans every deployment that has served it, so environment logs and metrics stay continuous across promotions: during and after a rollout, one view covers the outgoing and incoming deployments. Fetch them from any surface: **To view an environment's logs and metrics**: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar. 2. Select your model. 3. On the model overview, choose **Logs** or **View metrics** for the environment. You can also reach the same views from any deployment's logs or metrics page: choose the environment from the dropdown in the upper left. **To fetch environment logs**: Target the environment by name instead of a deployment ID: ```bash Command theme={"system"} baseten model environment logs --model-id --environment production --since 1h ``` ```txt Output theme={"system"} [2026-07-09 08:49:09]: (w44tt) Application shutdown complete. [2026-07-09 08:48:41]: Scaling down replicas due to inactivity [2026-07-09 08:34:09]: (67wjm) Terminated ... ``` The same windows and filters as deployment logs apply, including `--tail` for live streaming and `--output jsonl` for machine-readable output. **To fetch environment metrics**: ```bash Command theme={"system"} baseten model environment metrics --model-id --environment production ``` ```txt Output theme={"system"} METRIC QUANTILE STAT VALUE baseten_replicas_active 1 baseten_end_to_end_response_time_seconds 0.5 - baseten_end_to_end_response_time_seconds 0.9 - baseten_end_to_end_response_time_seconds 0.95 - baseten_end_to_end_response_time_seconds 0.99 - baseten_end_to_end_response_time_seconds avg - ``` The default is a current snapshot; pass `--mode summary` or `--mode series` with `--since` to aggregate over a window. In series mode the window splits at each promotion, so every point reflects the deployment serving the environment at that time. **To fetch environment logs**: ```bash Request theme={"system"} curl "https://api.baseten.co/v1/models/{model_id}/environments/production/logs" \ -H "Authorization: Bearer $BASETEN_API_KEY" ``` ```json Response theme={"system"} { "logs": [ { "timestamp": "1783612149447676900", "message": "Application shutdown complete.", "replica": "w44tt", "request_id": null, "level": "INFO" } ] } ``` **To fetch environment metrics**: ```bash Request theme={"system"} curl "https://api.baseten.co/v1/models/{model_id}/environments/production/metrics" \ -H "Authorization: Bearer $BASETEN_API_KEY" ``` ```json Response theme={"system"} { "start_epoch_millis": 1783968056212, "end_epoch_millis": 1783968056212, "mode": "CURRENT", "metric_descriptors": [ { "name": "baseten_replicas_active", "unit_hint": "COUNT", "kind": "GAUGE", "label_sets": [{}] }, ... ], "metric_values": [ { "start_epoch_millis": 1783968056212, "values": [[1.0]] } ] } ``` Both endpoints take the same windows and modes as their deployment counterparts. For more information, see [`environment logs`](/reference/cli/baseten/model-environment#logs) and [`environment metrics`](/reference/cli/baseten/model-environment#metrics), and the [environment logs](/reference/management-api/environments/get-environment-logs) and [environment metrics](/reference/management-api/environments/get-environment-metrics) (beta) endpoints. ## Next steps Check logs and metrics before and after a deployment change. * [Terminate a stuck replica](/troubleshooting/deployments#issue-a-single-replica-is-stuck-or-unhealthy) you found misbehaving in the logs. * [Scale a deployment](/deployment/manage/scaling) if metrics show sustained queueing or idle replicas. * [Metrics](/observability/metrics) for dashboards and metric definitions. # Manage deployments Source: https://docs.baseten.co/deployment/manage/overview Scale, promote, inspect, deactivate, and delete Baseten deployments from the dashboard, the Baseten CLI, or the Management API. Learn to operate deployments on Baseten. After you push a model, you'll scale it ahead of traffic, promote new versions, inspect logs and metrics, and deactivate or delete deployments you no longer need. You can manage your deployments using any of the following methods: * **[Dashboard](https://app.baseten.co)**: the Baseten web interface, for interactive changes and checking state. * **[Baseten CLI](/reference/cli/baseten/overview)**: terminal commands for scripting and automation. All commands except the `baseten truss` passthrough support `--output json` and `--jq` filtering. * **[Management API](/reference/management-api/overview)**: REST endpoints for managing models, deployments, and environments from your own code. * **[CI/CD](/deployment/ci-cd)**: automated deploys and promotions from GitHub Actions. ## Find your task * [Scale a deployment](/deployment/manage/scaling): change how much compute a serving deployment uses. Pre-scale before a traffic spike, scale to zero, or wake a scaled-to-zero deployment. * [Manage the deployment lifecycle](/deployment/manage/lifecycle): change whether and where a deployment serves. Promote, deactivate, activate, and delete deployments. * [Label deployments](/deployment/manage/labels): attach deployment metadata for ownership, source control, or CI automation. * [Terminate a stuck replica](/troubleshooting/deployments#issue-a-single-replica-is-stuck-or-unhealthy): remove one unhealthy replica and let the autoscaler replace it. * [Pull logs and metrics](/deployment/manage/logs-and-metrics): inspect a deployment or environment from the dashboard, Baseten CLI, or Management API. * [Download or inspect a deployment's config](#download-or-inspect-a-deployments-config): fetch a deployment's Truss config or source from the Baseten CLI. ## Before you start ### Authenticate Sign in with your Baseten account and create an [API key](/organization/api-keys): **To sign in**: 1. Go to [app.baseten.co](https://app.baseten.co). 2. Enter your email and choose **Continue**, or choose **Continue with Google** or **Continue with GitHub**. **To create an API key**: 1. In your [workspace](https://app.baseten.co), open **API keys** in your settings. 2. Choose **Create API key**. **To sign in**: ```bash Command theme={"system"} baseten auth login ``` ```txt Output theme={"system"} ? How would you like to authenticate? > Login with Baseten credentials (browser) Paste an API key Browser opened to authenticate... If it didn't open, visit: https://login.baseten.co/device?user_code=ABCD-1234 Verification code: ABCD-1234 ``` Complete the sign-in in the browser; the CLI stores a profile that authenticates later commands. For more information, see [`baseten auth login`](/reference/cli/baseten/auth#login). **To create an API key**: Create a personal key, tied to your account and its permissions, for local development and testing: ```bash Command theme={"system"} baseten org api-key create --type personal --name Create a team key, not tied to any one user and optionally scoped to specific models, for production and shared automation: ```bash Command theme={"system"} baseten org api-key create --type workspace-invoke --name For more information, see [`baseten org api-key`](/reference/cli/baseten/org-api-key). Commands also accept the key through the `BASETEN_API_KEY` environment variable instead of a profile. **To authenticate API requests**: The API doesn't sign in; every request carries an API key. Create your first key in the dashboard or CLI. 1. Set the key as an environment variable, or store it in your secret manager: ```bash macOS/Linux theme={"system"} export BASETEN_API_KEY="paste-your-api-key-here" ``` ```powershell Windows theme={"system"} setx BASETEN_API_KEY "paste-your-api-key-here" ``` 2. Pass the key in the `Authorization` header on every request: ```bash Request theme={"system"} curl "https://api.baseten.co/v1/models" \ -H "Authorization: Bearer $BASETEN_API_KEY" ``` ```json Response theme={"system"} { "models": [ { "id": "abc123", "name": "HelloWorld", "deployments_count": 1, "production_deployment_id": "def456", ... } ] } ``` **To create an API key with the API**: Once you have a first key, create more over the API. `type` takes `PERSONAL`, `WORKSPACE_MANAGE_ALL`, `WORKSPACE_INVOKE`, `WORKSPACE_EXPORT_METRICS`, or `WORKSPACE_MANAGE_API_KEYS`: ```bash Request theme={"system"} curl -X POST "https://api.baseten.co/v1/api_keys" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -H "Content-Type: application/json" \ -d '{"type": "PERSONAL", "name": " For more information, see the [create API key endpoint](/reference/management-api/api-keys/creates-an-api-key). For enterprise authentication, including identity-provider sign-in and SCIM user provisioning, see [SSO and SCIM](/organization/sso-and-scim) or [contact support](mailto:support@baseten.co) to enable it for your workspace. ### Find your model and deployment IDs Each deployment has an ID, associated with the model that owns it. Use the two IDs together to target every operation on these pages. Both appear in the model's page URL: Anatomy of the deployment page URL. In app.baseten.co/models/abc123/deployments/def456, abc123 is the model ID and def456 is the deployment ID. Anatomy of the deployment page URL. In app.baseten.co/models/abc123/deployments/def456, abc123 is the model ID and def456 is the deployment ID. **To find your IDs in the dashboard**: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar. 2. Select your model. 3. Choose **Copy ID** next to the model name. 4. Select the deployment under **Deployments**. The deployment ID is the last segment of the page URL. **To list your models and deployments**: 1. List your deployed models: ```bash Command theme={"system"} baseten model list ``` ```txt Output theme={"system"} ID NAME TEAM DEPLOYMENTS CREATED abc123 HelloWorld Baseten 1 2026-05-28T15:52:18Z ``` 2. Then list your model's deployments: ```bash Command theme={"system"} baseten model deployment list --model-id ``` ```txt Output theme={"system"} ID NAME ENVIRONMENT STATUS INSTANCE REPLICAS CREATED def456 deployment-1 production ACTIVE 1x2 - 1 vCPU, 2 GiB RAM 1 2026-05-28T15:52:19Z ``` Commands that take `--model-id` and `--deployment-id` also accept `--model-name` and `--deployment-name` to target by name instead. For more information, see [`model list`](/reference/cli/baseten/model#list) and [`deployment list`](/reference/cli/baseten/model-deployment#list). **To list your models and deployments**: 1. List your deployed models: ```bash Request theme={"system"} curl "https://api.baseten.co/v1/models" \ -H "Authorization: Bearer $BASETEN_API_KEY" ``` ```json Response theme={"system"} { "models": [ { "id": "abc123", "name": "HelloWorld", "deployments_count": 1, "production_deployment_id": "def456", ... } ] } ``` 2. Then list your model's deployments: ```bash Request theme={"system"} curl "https://api.baseten.co/v1/models/{model_id}/deployments" \ -H "Authorization: Bearer $BASETEN_API_KEY" ``` ```json Response theme={"system"} { "deployments": [ { "id": "def456", "name": "deployment-1", "model_id": "abc123", "is_production": true, "status": "ACTIVE", ... } ] } ``` For more information, see the [list models](/reference/management-api/models/gets-all-models) and [list deployments](/reference/management-api/deployments/gets-all-deployments-of-a-model) endpoints. ## Download or inspect a deployment's config Fetch a deployment's Truss config or source to inspect it locally. **To print a deployment's config.yaml**: ```bash Command theme={"system"} baseten model deployment config --model-id --deployment-id ``` **To download a deployment's source**: ```bash Command theme={"system"} baseten model deployment download --model-id --deployment-id --out-dir ./my-model ``` Use `--out-file` to save the source as a tar file instead of extracting it to a directory. Pass `--overwrite` to replace an existing file or non-empty directory. For more information, see [`deployment config`](/reference/cli/baseten/model-deployment#config) and [`deployment download`](/reference/cli/baseten/model-deployment#download). ## Next steps For the concepts behind these operations, see [Deployments](/deployment/deployments) and [Environments](/deployment/environments). For incremental promotion, see [Rolling deployments](/deployment/rolling-deployments). # Scale a deployment Source: https://docs.baseten.co/deployment/manage/scaling Update autoscaling settings and scale to zero through the dashboard or Management API, and wake deployments through the dashboard or Inference API. Use this page to update autoscaling settings on a live deployment, scale it to zero, or wake it. To choose the values, start with the [autoscaling overview](/deployment/autoscaling/overview) and match the settings to your [traffic patterns](/deployment/autoscaling/traffic-patterns). ## Update autoscaling settings Update autoscaling settings in place to pre-scale for a known traffic spike, raise a replica ceiling, or tune scale-down behavior. The change applies to the running deployment; replicas adjust without a new deploy. **To update autoscaling settings**: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar. 2. Select your model. 3. On the model's overview, choose **Configure** under the environment's **Replicas**. 4. Adjust your settings: * **Replicas**: the minimum and maximum replica count. Baseten scales within this range based on traffic. * **Autoscaling window**: how far back the autoscaler looks when averaging traffic for scaling decisions. * **Scale down delay**: how long the autoscaler waits after traffic drops before removing replicas. * **Max scale down rate**: the largest percentage of active replicas the autoscaler removes in a single scale-down step. * **Concurrency target**: the number of concurrent requests each replica handles before the autoscaler adds another. * **Target utilization percentage**: the share of the concurrency target at which scaling triggers. 5. Choose **Update** to apply the changes. The same dialog is in the environment card's **⋯** menu as **Configure autoscaling**. **To update autoscaling settings**: Pass only the flags you want to change; every other setting is left alone. The update applies asynchronously: ```bash Command theme={"system"} baseten model deployment update-autoscaling \ --model-id --deployment-id \ --min-replica 2 --max-replica 8 ``` ```text Output theme={"system"} ACCEPTED Your request to update autoscaling settings has been accepted. Query for deployment 's status to see when the updates have been applied. ``` See [`baseten model deployment update-autoscaling`](/reference/cli/baseten/model-deployment#update-autoscaling) for every flag. **To update autoscaling settings**: Use the `autoscaling_settings` endpoint to apply autoscaling updates. It accepts any subset of the fields, so send only the ones you're changing; the update applies asynchronously: ```bash Request theme={"system"} curl -X PATCH "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/autoscaling_settings" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "min_replica": 2, "max_replica": 8, "autoscaling_window": 60, "scale_down_delay": 900, "concurrency_target": 2, "target_utilization_percentage": 70 }' ``` ```json Response theme={"system"} { "status": "ACCEPTED", "message": "Your request to update autoscaling settings has been accepted. Query for deployment {deployment_id}'s status to see when the updates have been applied." } ``` Autoscaling updates apply asynchronously. Verify the deployment after you submit a change: **To verify your settings**: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar. 2. Select your model. 3. Select the deployment under **Deployments**. 4. Review the current values on the **Autoscaling settings** card: replicas, autoscaling window, scale down delay, concurrency target, and target utilization. **To verify your settings**: ```bash Command theme={"system"} baseten model deployment describe --model-id --deployment-id --jq '.autoscaling_settings' ``` ```json Output theme={"system"} { "autoscaling_window": 60, "concurrency_target": 2, "max_replica": 8, "min_replica": 2, "scale_down_delay": 900, "target_utilization_percentage": 70 } ``` **To verify your settings**: ```bash Request theme={"system"} curl "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}" \ -H "Authorization: Bearer $BASETEN_API_KEY" ``` ```json Response theme={"system"} { "id": "def456", "autoscaling_settings": { "min_replica": 2, "max_replica": 8, "autoscaling_window": 60, "scale_down_delay": 900, "concurrency_target": 2, "target_utilization_percentage": 70 }, ... } ``` For more information, see the [update autoscaling settings endpoint](/reference/management-api/deployments/autoscaling/updates-a-deployments-autoscaling-settings). To update settings on whatever deployment an environment currently serves, use the [environment settings endpoint](/reference/management-api/environments/update-an-environments-settings) instead. ## Scale back down After a spike passes, lower `min_replica` back to its normal floor with the same [update procedure](#update-autoscaling-settings). Two settings control how fast replicas drain: * **Scale down delay** (`scale_down_delay`): how long the autoscaler waits after traffic drops before removing replicas. * **Max scale down rate**: the largest percentage of active replicas removed in a single step. Set it in the dashboard's **Configure autoscaling** dialog. For example, drop the floor back to one replica and keep a 15-minute drain delay: ```bash Request theme={"system"} curl -X PATCH "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/autoscaling_settings" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -H "Content-Type: application/json" \ -d '{"min_replica": 1, "scale_down_delay": 900}' ``` ```json Response theme={"system"} { "status": "ACCEPTED", "message": "Your request to update autoscaling settings has been accepted. Query for deployment {deployment_id}'s status to see when the updates have been applied." } ``` Replicas above the new floor drain gradually; traffic keeps flowing to the replicas that remain. ## Scale to zero Set `min_replica` to `0` to let an idle deployment release all its replicas and stop billing for compute. Scaling to zero adds a [cold start](/deployment/autoscaling/cold-starts) when traffic returns. Synchronous requests can wait while a replica starts. Use scale-to-zero only when your workload can tolerate that wait. **To scale to zero**: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar. 2. Select your model. 3. On the model's overview, choose **Configure** under the environment's **Replicas**. 4. Set **Min** replicas to `0`. 5. Choose **Update** to apply the changes. **To scale to zero**: ```bash Request theme={"system"} curl -X PATCH "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/autoscaling_settings" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -H "Content-Type: application/json" \ -d '{"min_replica": 0}' ``` ```json Response theme={"system"} { "status": "ACCEPTED", "message": "Your request to update autoscaling settings has been accepted. Query for deployment {deployment_id}'s status to see when the updates have been applied." } ``` The model's endpoint stays live while the deployment has no replicas. To stop serving entirely, [deactivate the deployment](/deployment/manage/lifecycle#deactivate-a-deployment) instead. ## Wake a scaled-to-zero deployment Wake a scaled-to-zero deployment before you need it, for example ahead of a demo or a batch job, so traffic avoids a cold-start wait or capacity rejection. **To wake a deployment**: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar. 2. Select your model. 3. Select the deployment under **Deployments**. 4. Choose **Wake deployment**. The environment card has the same action as **Wake**. **To wake a deployment**: ```bash Request theme={"system"} curl -i -X POST "https://model-{model_id}.api.baseten.co/deployment/{deployment_id}/wake" \ -H "Authorization: Bearer $BASETEN_API_KEY" ``` ```txt Response theme={"system"} HTTP/2 202 content-length: 0 ``` The deployment starts a replica in the background, moving from `SCALED_TO_ZERO` through `WAKING_UP` to `ACTIVE`; poll [`baseten model deployment describe`](/reference/cli/baseten/model-deployment#describe) until `status` is `ACTIVE`. A woken deployment with `min_replica: 0` scales back down after `scale_down_delay` if no requests arrive, so wake it close to when you need it, or [raise `min_replica`](#update-autoscaling-settings) to hold it warm. ## Next steps Scaling changes take effect on the running deployment, so pair them with a quick status check before and after. * [Autoscaling overview](/deployment/autoscaling/overview) to choose replica counts, concurrency targets, and scale-down behavior. * [Manage the deployment lifecycle](/deployment/manage/lifecycle) to deactivate, promote, or delete deployments. * [Pull logs and metrics](/deployment/manage/logs-and-metrics) to confirm how a scaling change lands. # Regional environments Source: https://docs.baseten.co/deployment/regional-environments Constrain replicas and route inference requests within a designated geographic region to meet data-residency requirements. Regional environments constrain deployment replicas to workload planes within a designated geographic region and provide a region-specific inference endpoint. Use them when your architecture requires region-specific placement and routing without separate models for each region. Regional environments require initial configuration by Baseten. [Contact support](mailto:support@baseten.co) to confirm availability for the region you need and configure regional restrictions for your environments. ## How regional environments work Regional environments build on [environments](/deployment/environments) and [restricted environments](/organization/restricted-environments) to add region-level placement and routing controls. When Baseten configures regional restrictions for an environment, two things happen: 1. **Replicas are constrained** to workload planes within the designated region. Deployments promoted to that environment only run in the allowed region. 2. **A regional inference endpoint** becomes available that routes requests directly to the region-specific workload plane. ### Compare regional and standard endpoints Regional endpoints use a different URL format that maps directly to a region-specific workload plane: | Endpoint type | URL format | Required-endpoint behavior | | :------------ | :------------------------------------------------------------------------ | :---------------------------------------- | | Standard | `https://model-{model_id}.api.baseten.co/environments/{env_name}/predict` | The request returns a `403` response. | | Regional | `https://model-{model_id}-{env_name}.api.baseten.co/predict` | Baseten routes the request to the region. | If an environment requires a regional endpoint, requests to its standard endpoint return a `403` response. Update your calling code to use the regional endpoint. ## Set up regional environments To set up regional environments: 1. Create environments with region-specific names (for example, `prod-us`, `prod-eu`, `staging-eu`). Use [restricted environments](/organization/restricted-environments) to control access. 2. [Contact Baseten support](mailto:support@baseten.co) to configure the required region and endpoint restrictions for your environments. 3. Update your calling code to use the regional endpoint format: `https://model-{model_id}-{env_name}.api.baseten.co/predict`. ### Environment naming requirements Environment names used with regional environments must be valid Domain Name System (DNS) subdomain labels: * Lowercase alphanumeric characters and hyphens only. * Can't start or end with a hyphen. * Maximum 40 characters. * Baseten reserves `development` and `region`. * Baseten reserves names that start with `region-`. Regional environments apply across all models and Chains in a team. If you name an environment `prod-us` on one model or Chain, using `prod-us` on another model or Chain in the same team applies the same regional restrictions. ## Deploy to regional environments Deploy and promote to regional environments the same way as standard environments: ```sh theme={"system"} baseten model push --environment prod-us --region ``` For the full flag set, see [`model push`](/reference/cli/baseten/model#push). ```sh theme={"system"} truss push --environment prod-us ``` Replicas spin up only in workload planes within the allowed region. ### Promotion behavior When you promote a deployment to a regional environment, Baseten places its replicas in the configured region. This placement change can require a redeployment. ## Regional endpoint formats Regional endpoints embed the environment name in the hostname instead of the URL path: Call a model with `/predict` or `/async_predict`: ```text theme={"system"} https://model-{model_id}-{env_name}.api.baseten.co/predict ``` Call a Chain with `/run_remote` or `/async_run_remote`: ```text theme={"system"} https://chain-{chain_id}-{env_name}.api.baseten.co/run_remote ``` Connect to a model or Chain with `/websocket`: ```text theme={"system"} wss://model-{model_id}-{env_name}.api.baseten.co/websocket wss://chain-{chain_id}-{env_name}.api.baseten.co/websocket ``` Connect to a model through the `grpc.api.baseten.co` subdomain: ```text theme={"system"} model-{model_id}-{env_name}.grpc.api.baseten.co:443 ``` The regional endpoint URL appears in the model's API endpoint section in the Baseten dashboard after Baseten configures the environment for your team. ### Request restrictions Regional endpoints derive the environment from the hostname. Don't add `/environments/`, `/production/`, or `/deployment/` to a regional endpoint path. For gRPC, don't set the `x-baseten-environment` or `x-baseten-deployment` metadata headers. ## Call a regional endpoint Regional endpoints accept the same request format as standard predict endpoints: **To call a regional endpoint**: Create an `httpx.Client` with the regional endpoint as the `base_url`. Reuse the client across requests for connection pooling. See [Configure HTTP clients](/inference/http-client-configuration) for recommended timeout and pool settings. ```python predict.py theme={"system"} import httpx import os model_id = "" env_name = "prod-us" client = httpx.Client( base_url=f"https://model-{model_id}-{env_name}.api.baseten.co", headers={"Authorization": f"Bearer {os.environ['BASETEN_API_KEY']}"}, ) response = client.post("/predict", json={"prompt": "Hello, world!"}) print(response.json()) ``` **To call a regional endpoint**: Send a POST request to the regional endpoint with your API key in the `Authorization` header: ```sh Request theme={"system"} curl -X POST https://model-{model_id}-{env_name}.api.baseten.co/predict \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -H 'Content-Type: application/json' \ -d '{"prompt": "Hello, world!"}' ``` **To call a regional endpoint**: Use the built-in `fetch` API to call the regional endpoint. Replace `modelId` and `envName` with your model ID and environment name: ```javascript predict.js theme={"system"} const modelId = ""; const envName = "prod-us"; const resp = await fetch( `https://model-${modelId}-${envName}.api.baseten.co/predict`, { method: "POST", headers: { Authorization: `Bearer ${process.env.BASETEN_API_KEY}`, "Content-Type": "application/json", }, body: JSON.stringify({ prompt: "Hello, world!" }), } ); const data = await resp.json(); console.log(data); ``` # Resources Source: https://docs.baseten.co/deployment/resources Select and configure CPU, GPU, memory, and multi-node resources for model deployments. Every deployed model runs on an **instance**, a dedicated set of CPU, memory, and optional GPU resources. Choose an instance that has enough memory and compute for the model without provisioning unused capacity. * Insufficient resources can cause slow inference, out-of-memory errors, or failed deployments. * Excess resources increase cost without improving performance. Baseten dashboard showing CPU, memory, and GPU resource settings for a deployment ## Instance type resource components * **Instance:** The hardware allocated to an inference replica. * **Node:** A compute server that provides the instance's CPU, RAM, GPUs, and VRAM. * **vCPU:** Virtual CPU cores for general computation. * **RAM:** Memory available to the CPU. * **GPU:** Hardware for accelerated model execution. * **VRAM:** Memory available to the GPU. ## Configure model resources Define resources in `config.yaml` before deployment, or update a published deployment through the Baseten dashboard. ### Define resources in Truss Define resource requirements in [`config.yaml`](/development/model/configuration) before running `baseten model push`. The push creates one of two deployment types: * **Published deployment:** `baseten model push` creates a published deployment with the resources in `config.yaml`. * **Development deployment:** `baseten model push --watch` replaces the development deployment with the specified resources and watches for changes. Use [`baseten model watch`](/development/model/deploy-and-iterate) to resume watching it. You can also push straight to a [custom environment](/deployment/environments), such as staging, with `baseten model push --environment `. To promote an existing deployment to production, use `baseten model deployment promote`. The Truss CLI combines both steps in `truss push --promote`. Changes to `config.yaml` only affect new deployments. To update resources on an existing published deployment, edit resources in the [Baseten dashboard](#update-resources-in-the-baseten-dashboard). You can configure resources in two ways: #### Individual resource fields ```yaml config.yaml theme={"system"} resources: accelerator: L4 cpu: "4" memory: 16Gi ``` Baseten provisions the smallest instance that meets the specified constraints: * `cpu: "3"` or `cpu: "4"` maps to a 4-core instance. * Values from `cpu: "5"` through `cpu: "8"` map to an 8-core instance. `Gi` in `resources.memory` means gibibytes rather than gigabytes. #### Exact instance type An instance type is the SKU for a specific hardware configuration. When you set individual fields such as `cpu` and `accelerator`, Baseten selects the smallest matching instance. Set `instance_type` to choose an exact SKU. Use `instance_type` when you: * Know the exact hardware configuration you need. * Want to ensure consistent instance selection across deployments. * Are following a recommendation for a specific model (for example, "use an L4 with 4 vCPUs and 16 GiB RAM"). ```yaml config.yaml theme={"system"} resources: instance_type: "L4:4x16" ``` The format encodes the hardware specs. For example, `L4:4x16` means an L4 GPU with 4 vCPUs and 16 GiB of RAM. Naming conventions vary by GPU family, so copy the exact instance type from the [instance type reference](#instance-type-reference). When `instance_type` is specified, other resource fields (`cpu`, `memory`, `accelerator`, `use_gpu`) are ignored. ### Update resources in the Baseten dashboard Update resources on an existing published deployment through the Baseten dashboard. Changing the instance type creates a copy of the deployment on the selected hardware. For a list of available instance types, see the [instance type reference](/deployment/resources#instance-type-reference). ### Multi-node deployments For models that don't fit on a single node, set `node_count` in `resources` to provision multiple identical nodes for one deployment. Each node gets the resources you specify, and Baseten connects the nodes with high-speed InfiniBand for inter-node communication. ```yaml config.yaml theme={"system"} resources: accelerator: H100:8 node_count: 2 ``` Multi-node inference is typically used with TensorRT-LLM's [v2 inference stack](/engines/engine-builder-llm/overview), which supports MoE and multi-node setups. Set `trt_llm.inference_stack: v2` in `config.yaml` when compiling the engine for a multi-node deployment. ## Instance type reference Use the following tables to compare specifications and prices for available instance types. ### CPU-only instances CPU-only instances suit workloads that do not require GPU acceleration. * **Starts at**: \$0.00058/min * **Best for**: Transformers pipelines, small QA models, text embeddings | Instance | \$/min | vCPU | RAM | | -------- | --------- | ---- | ------ | | `1x2` | \$0.00058 | 1 | 2 GiB | | `1x4` | \$0.00086 | 1 | 4 GiB | | `2x8` | \$0.00173 | 2 | 8 GiB | | `4x16` | \$0.00346 | 4 | 16 GiB | | `8x32` | \$0.00691 | 8 | 32 GiB | | `16x64` | \$0.01382 | 16 | 64 GiB | To select a CPU-only instance, use the bare `x` SKU (for example, `instance_type: "4x16"`). **Example workloads:** * `1x2`: Text classification (for example, Truss quickstart) * `4x16`: LayoutLM Document QA * `4x16+`: Sentence Transformers embeddings on larger corpora ### GPU instances Accelerated inference for LLMs, diffusion models, and Whisper. | Instance | \$/min | vCPU | RAM | GPU | VRAM | | ---------------- | --------- | ---- | -------- | ---------------------- | -------- | | `T4x4x16` | \$0.01052 | 4 | 16 GiB | 1 NVIDIA T4 | 16 GiB | | `T4x8x32` | \$0.01504 | 8 | 32 GiB | 1 NVIDIA T4 | 16 GiB | | `T4x16x64` | \$0.02408 | 16 | 64 GiB | 1 NVIDIA T4 | 16 GiB | | `T4:2x24x96` | \$0.03912 | 24 | 96 GiB | 2 NVIDIA T4s | 32 GiB | | `T4:4x48x192` | \$0.07824 | 48 | 192 GiB | 4 NVIDIA T4s | 64 GiB | | `L4:4x16` | \$0.01414 | 4 | 16 GiB | 1 NVIDIA L4 | 24 GiB | | `L4:2x24x96` | \$0.04002 | 24 | 96 GiB | 2 NVIDIA L4s | 48 GiB | | `L4:4x48x192` | \$0.08003 | 48 | 192 GiB | 4 NVIDIA L4s | 96 GiB | | `A10Gx4x16` | \$0.02012 | 4 | 16 GiB | 1 NVIDIA A10G | 24 GiB | | `A10Gx8x32` | \$0.02424 | 8 | 32 GiB | 1 NVIDIA A10G | 24 GiB | | `A10Gx16x64` | \$0.03248 | 16 | 64 GiB | 1 NVIDIA A10G | 24 GiB | | `A10G:2x24x96` | \$0.05672 | 24 | 94 GiB | 2 NVIDIA A10Gs | 48 GiB | | `A10G:4x48x192` | \$0.11344 | 48 | 188 GiB | 4 NVIDIA A10Gs | 96 GiB | | `A10G:8x192x768` | \$0.32576 | 192 | 750 GiB | 8 NVIDIA A10Gs | 192 GiB | | `A100:12x144` | \$0.06667 | 12 | 144 GiB | 1 NVIDIA A100 | 80 GiB | | `A100:2x24x288` | \$0.13334 | 24 | 288 GiB | 2 NVIDIA A100s | 160 GiB | | `A100:3x36x432` | \$0.20000 | 36 | 432 GiB | 3 NVIDIA A100s | 240 GiB | | `A100:4x48x576` | \$0.26668 | 48 | 576 GiB | 4 NVIDIA A100s | 320 GiB | | `A100:5x60x720` | \$0.33333 | 60 | 720 GiB | 5 NVIDIA A100s | 400 GiB | | `A100:6x72x864` | \$0.40000 | 72 | 864 GiB | 6 NVIDIA A100s | 480 GiB | | `A100:7x84x1008` | \$0.46667 | 84 | 1008 GiB | 7 NVIDIA A100s | 560 GiB | | `A100:8x96x1152` | \$0.53333 | 96 | 1152 GiB | 8 NVIDIA A100s | 640 GiB | | `H100` | \$0.10833 | 16 | 118 GiB | 1 NVIDIA H100 | 80 GiB | | `H100:2` | \$0.21666 | 32 | 236 GiB | 2 NVIDIA H100s | 160 GiB | | `H100:4` | \$0.43332 | 64 | 472 GiB | 4 NVIDIA H100s | 320 GiB | | `H100:8` | \$0.86664 | 128 | 944 GiB | 8 NVIDIA H100s | 640 GiB | | `H100MIG` | \$0.06250 | 8 | 59 GiB | Fractional NVIDIA H100 | 40 GiB | | `H200` | \$0.12500 | 16 | 200 GiB | 1 NVIDIA H200 | 141 GiB | | `H200:2` | \$0.25000 | 32 | 400 GiB | 2 NVIDIA H200s | 282 GiB | | `H200:4` | \$0.50000 | 64 | 800 GiB | 4 NVIDIA H200s | 564 GiB | | `H200:8` | \$1.00000 | 128 | 1600 GiB | 8 NVIDIA H200s | 1128 GiB | | `B200` | \$0.16633 | 16 | 224 GiB | 1 NVIDIA B200 | 180 GiB | | `B200:2` | \$0.33266 | 32 | 448 GiB | 2 NVIDIA B200s | 360 GiB | | `B200:4` | \$0.66532 | 64 | 896 GiB | 4 NVIDIA B200s | 720 GiB | | `B200:8` | \$1.33064 | 128 | 1792 GiB | 8 NVIDIA B200s | 1440 GiB | | `RTX-PRO-6000` | \$0.06667 | 16 | 116 GiB | 1 NVIDIA RTX-PRO-6000 | 96 GiB | | `RTX-PRO-6000:2` | \$0.13334 | 32 | 233 GiB | 2 NVIDIA RTX-PRO-6000s | 192 GiB | | `RTX-PRO-6000:4` | \$0.26668 | 64 | 466 GiB | 4 NVIDIA RTX-PRO-6000s | 384 GiB | | `RTX-PRO-6000:8` | \$0.53336 | 128 | 931 GiB | 8 NVIDIA RTX-PRO-6000s | 768 GiB | To select a GPU instance with `instance_type`: * **Single L4 or A100**: `:x` (for example, `"L4:4x16"`). * **Single T4 or A10G**: `xx`, with no colon (for example, `"T4x4x16"`, `"A10Gx8x32"`). * **Multi-GPU**: `:xx` (for example, `"A100:2x24x288"`). * **H100/H200/B200/RTX-PRO-6000**: `` or `:` (for example, `"H100:2"`, `"RTX-PRO-6000:4"`). * **Fractional H100**: `"H100MIG"`. Naming is not uniform across GPU families, so copy the exact SKU from the tables above. ### GPU details and workloads #### T4 Turing-series GPU * 2,560 CUDA / 320 Tensor cores * 16 GiB VRAM * **Best for:** Whisper, small LLMs like StableLM 3B #### L4 Ada Lovelace-series GPU * 7,680 CUDA / 240 Tensor cores * 24 GiB VRAM, 300 GiB/s * 121 TFLOPS (fp16) * **Best for:** Small LLMs and diffusion models such as Stable Diffusion XL. #### A10G Ampere-series GPU * 9,216 CUDA / 288 Tensor cores * 24 GiB VRAM, 600 GiB/s * 70 TFLOPS (fp16) * **Best for**: Mistral 7B, Whisper, Stable Diffusion/SDXL #### A100 Ampere-series GPU * 6,912 CUDA / 432 Tensor cores * 80 GiB VRAM, 1.94 TB/s * 312 TFLOPS (fp16) * **Best for**: Mixtral, Llama 2 70B (2 A100s), Falcon 180B (5 A100s), SDXL #### H100 Hopper-series GPU * 16,896 CUDA / 640 Tensor cores * 80 GiB VRAM, 3.35 TB/s * 990 TFLOPS (fp16) * **Best for**: Mixtral 8x7B, Llama 2 70B (2xH100), SDXL #### H100MIG Fractional H100 (3/7 compute, ½ memory) * 7,242 CUDA cores, 40 GiB VRAM * 1.675 TB/s bandwidth * **Best for**: Efficient LLM inference at lower cost than A100 #### RTX Pro 6000 Blackwell-series GPU * 96 GiB VRAM * **Best for**: vision-language models and mid-size LLMs at lower cost than a datacenter GPU # Rolling deployments Source: https://docs.baseten.co/deployment/rolling-deployments Gradually shift traffic to a new deployment with replica-based rolling deployments. Rolling deployments replace replicas incrementally when promoting a deployment to an environment. Instead of swapping all traffic at once, rolling deployments scale up the candidate deployment, shift traffic toward it, and scale down the previous deployment in controlled steps. Autoscaling continues throughout the rollout for environments where `min_replica < max_replica`, so Baseten adjusts capacity across the previous and candidate deployments as demand changes. Use rolling deployments when you need zero-downtime updates with the ability to pause, cancel, or force-complete the deployment at any point. Rolling deployments are not supported for [Chains](/chains/overview). This feature is available for individual model deployments only. ## How rolling deployments work A rolling deployment follows a repeating three-step cycle: 1. **Scale up** candidate deployment replicas by the configured percentage. 2. **Shift traffic** to match the updated replica allocation. 3. **Scale down** the previous deployment replicas by the same percentage. This cycle repeats until all traffic and replicas run on the candidate deployment, at which point it becomes the active deployment in the environment. Scaling down the previous deployment doesn't interrupt requests it's already serving. Replicas drain: they stop accepting new requests and finish in-flight ones before terminating. See [Requests during a promotion](/deployment/autoscaling/request-lifecycle#requests-during-a-promotion) for the draining behavior and its limits. The following diagram shows this cycle in action. The tab strip mirrors the promotion lifecycle: a promotion enters `RELEASING` when it starts, sits in `RAMPING_UP` while replicas scale and traffic shifts, can pause as `PAUSED`, and lands at `SUCCEEDED` once the candidate serves all traffic. Select any status to freeze the simulation on that stage, then select it again to resume. Adjust the values and choose **Apply** to restart the simulation with your configuration. The simulator accepts `replica_overhead_percent` values up to 60% so the replica rows remain readable. The API accepts values from 0% through 500%. ### Provisioning modes Rolling deployments support two mutually exclusive provisioning modes: * `max_surge_percent`: Scales up candidate replicas before scaling down previous replicas. * `max_unavailable_percent`: Scales down previous replicas before scaling up candidate replicas. Set exactly one mode to a non-zero value and the other to `0`. ## Enable rolling deployments Rolling deployments are off by default. Enable them per environment in its promotion settings: **To enable rolling deployments**: ```bash Command theme={"system"} baseten model environment update-promotion --environment production \ --rolling-deploy true \ --max-surge-percent 25 \ --max-unavailable-percent 0 \ --stabilization-time-seconds 60 \ --replica-overhead-percent 0 ``` ```txt Output theme={"system"} Updated promotion settings for environment production ``` For the full flag set, see [`model environment update-promotion`](/reference/cli/baseten/model-environment#update-promotion). **To enable rolling deployments**: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar. 2. Select your model. 3. On the model's overview, open the environment card's **⋯** menu. 4. Choose **Configure promotion**. 5. Turn on **Rolling deploys**. 6. Choose **Update** to apply the changes. **To enable rolling deployments**: ```bash Request theme={"system"} curl -X PATCH "https://api.baseten.co/v1/models/{model_id}/environments/production" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "promotion_settings": { "rolling_deploy": true, "rolling_deploy_config": { "max_surge_percent": 25, "max_unavailable_percent": 0, "stabilization_time_seconds": 60, "replica_overhead_percent": 0 } } }' ``` ```json Response theme={"system"} { "status": "ACCEPTED", "message": "Your request to update environment settings is complete" } ``` Once rolling deployments are enabled, any subsequent [promotion to the environment](/deployment/manage/lifecycle#promote-to-an-environment) uses the rolling deployment workflow. ## Configuration reference Configure rolling deployments through the `rolling_deploy_config` object in the environment's `promotion_settings`. Percentage of additional replicas to provision during each step. Set to `0` to use max unavailable mode instead. **Range:** 0-100 Percentage of replicas that can be unavailable during each step. Set to `0` to use max surge mode instead. **Range:** 0-50 Seconds to wait after each traffic shift before proceeding to the next step. Use this to monitor metrics between steps. **Range:** 0-3600 Percentage of additional replicas to pre-provision on the current deployment before the rolling deployment starts. Useful for environments without autoscaling (`min_replica == max_replica`) or as a buffer for anticipated traffic spikes during the rollout. **Range:** 0-500 Additional promotion settings configured at the `promotion_settings` level: Enables rolling deployments for the environment. ## Deployment statuses The `in_progress_promotion` field on the [environment detail endpoint](/reference/management-api/environments/get-an-environments-details) tracks the current state of a rolling deployment. It's separate from each deployment's own `status`. | Status | Description | | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `RELEASING` | Candidate deployment is building and initializing replicas. | | `RAMPING_UP` | Scaling up candidate replicas and shifting traffic. | | `PAUSED` | Rolling deployment is paused at its current traffic split. No further promotion steps run, but in-flight replica changes and autoscaling continue. | | `RAMPING_DOWN` | Graceful cancel in progress. Traffic is shifting back to the previous deployment. | | `SUCCEEDED` | Rolling deployment completed. The candidate is now the active deployment. | | `FAILED` | Rolling deployment failed. Traffic remains on the previous deployment. | | `CANCELED` | Rolling deployment was canceled. Traffic returned to the previous deployment. | The `in_progress_promotion` object also includes `percent_traffic_to_new_version`, which reports the current percentage of traffic routed to the candidate deployment. To watch metrics through a rollout, use the [environment metrics endpoint](/reference/management-api/environments/get-environment-metrics) (beta): it aggregates both deployments, and in series mode it splits the window at each promotion so every point reflects the deployments serving at that time. ## Deployment control actions Pause, resume, and force roll forward act on the rolling deployment between steps, not immediately. Replica changes already in progress finish before the action takes effect, so the rolling deployment can keep scaling for a short time after you trigger the action. For example, if the candidate deployment is at 20% traffic and has just been told to scale from 2 to 4 replicas, choosing **Pause** lets the candidate finish scaling to 4 replicas. The traffic split stays pinned at 20% until you resume. ### Pause Pause the rolling deployment to inspect metrics or logs before proceeding: **To pause a rolling deployment**: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar. 2. Select your model. 3. On the model's overview, find the environment's promotion banner. 4. Choose **Pause**. **To pause a rolling deployment**: ```bash Request theme={"system"} curl -X POST "https://api.baseten.co/v1/models/{model_id}/environments/production/pause_promotion" \ -H "Authorization: Bearer $BASETEN_API_KEY" ``` ```json Response theme={"system"} {"success": true} ``` ### Resume Resume a paused rolling deployment from where it left off: **To resume a rolling deployment**: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar. 2. Select your model. 3. On the model's overview, find the environment's promotion banner. 4. Choose **Resume**. **To resume a rolling deployment**: ```bash Request theme={"system"} curl -X POST "https://api.baseten.co/v1/models/{model_id}/environments/production/resume_promotion" \ -H "Authorization: Bearer $BASETEN_API_KEY" ``` ```json Response theme={"system"} {"success": true} ``` ### Cancel Gracefully cancel the rolling deployment. Traffic ramps back to the previous deployment and candidate replicas scale down: **To cancel a rolling deployment**: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar. 2. Select your model. 3. On the model's overview, find the environment's promotion banner. 4. Choose **Cancel**. **To cancel a rolling deployment**: ```bash Request theme={"system"} curl -X POST "https://api.baseten.co/v1/models/{model_id}/environments/production/cancel_promotion" \ -H "Authorization: Bearer $BASETEN_API_KEY" ``` ```json Response theme={"system"} {"status": "RAMPING_DOWN", "message": "..."} ``` Returns a `status` of `CANCELED` (instant cancel for non-rolling deployments) or `RAMPING_DOWN` (graceful rollback for rolling deployments). ### Force cancel Immediately cancel the rolling deployment and return all traffic to the previous deployment when you can't wait for the graceful ramp-down. Force canceling may cause brief service disruption if the previous deployment is under-provisioned. **To force cancel a rolling deployment**: ```bash Request theme={"system"} curl -X POST "https://api.baseten.co/v1/models/{model_id}/environments/production/force_cancel_promotion" \ -H "Authorization: Bearer $BASETEN_API_KEY" ``` ```json Response theme={"system"} {"success": true} ``` ### Force roll forward Immediately complete the rolling deployment, shifting all traffic to the candidate deployment. This works even while the deployment is rolling back: Force rolling forward may promote an under-provisioned deployment if the candidate has not finished scaling up. **To force roll forward**: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar. 2. Select your model. 3. On the model's overview, find the environment's promotion banner. 4. Choose **Force promote**. **To force roll forward**: ```bash Request theme={"system"} curl -X POST "https://api.baseten.co/v1/models/{model_id}/environments/production/force_roll_forward_promotion" \ -H "Authorization: Bearer $BASETEN_API_KEY" ``` ```json Response theme={"system"} {"success": true} ``` ## Autoscaling during rolling deployments For environments configured with autoscaling (`min_replica < max_replica`), Baseten continues to adjust capacity during a rolling deployment based on combined demand. The autoscaler can add or remove replicas from either deployment as traffic changes and the rollout advances. Replica allocation accounts for rollout state as well as traffic, so the replica split doesn't necessarily match the traffic split. A few constraints apply during the rolling deployment: * Autoscaling adds and removes replicas throughout the rollout to track combined demand. Each deployment that is still part of the rollout keeps at least one replica. * Capacity management continues during a `PAUSED` rolling deployment. Pausing stops the traffic shift, not capacity management. If demand changes while paused, both deployments still scale up or down. ## Dynamic replica admission Rolling deployments adapt to candidate replicas as they become ready. Rather than assuming a full batch of replicas will be available immediately, Baseten adjusts the rollout based on live capacity. For example, with 100 previous replicas and `max_unavailable_percent` set to `25`, Baseten requests 25 new replicas. If only 5 become ready, Baseten only removes 5 previous replicas to stay within your unavailable limit: ```text theme={"system"} Max unavailable 25% Requested: Previous 75 replicas Candidate 25 requested Actually ready: Previous 75 replicas Candidate 5 ready Next adjustment: Previous 70 replicas Candidate 5 ready ``` The same adaptive behavior applies to `max_surge_percent`. With 100 previous replicas and a 25% surge limit, if only 5 of the 25 requested candidate replicas become ready, Baseten scales down 5 previous replicas before requesting the next batch. This ensures the rollout progresses based on actual ready capacity. ```text theme={"system"} Max surge 25% Requested: Previous 100 replicas Candidate 25 requested Actually ready: Previous 100 replicas Candidate 5 ready Next adjustment: Previous 95 replicas Candidate 5 ready ``` In both modes, rollouts continue from live, ready capacity to ensure your environment remains stable throughout the transition. ## Environments without autoscaling Environments where `min_replica == max_replica` have no autoscaling configured, so replica counts stay pinned during the rolling deployment. To pre-provision additional headroom for traffic spikes, set `replica_overhead_percent` to add replicas to the previous deployment before any traffic shifts. Use `stabilization_time_seconds` to wait between steps and monitor metrics before the next traffic shift. ## Deployment cleanup When a promotion completes, the `promotion_cleanup_strategy` setting controls what happens to the deployment it replaced. The strategy applies to every promotion into the environment, not only to rolling ones. * `SCALE_TO_ZERO`: Sets the previous deployment's minimum replicas to 0 and leaves its maximum unchanged, so the autoscaler drains it over the scale-down delay. The deployment stays `ACTIVE` and scales back up on demand. This is the default. * `KEEP`: Leaves the previous deployment running at its current replica count. * `DEACTIVATE`: Deactivates the previous deployment, so it stops serving traffic and releases its replicas. Both `SCALE_TO_ZERO` and `DEACTIVATE` set the previous deployment's minimum replicas to 0, and that change persists. A deployment pinned to two minimum replicas keeps a minimum of 0 after the promotion, so set its [autoscaling settings](/deployment/manage/scaling) again before you promote it back into service. Only `DEACTIVATE` releases the previous deployment's slot against your workspace's [deployment limit](/troubleshooting/deployments#issue-youve-reached-the-maximum-number-of-deployed-models). Under `SCALE_TO_ZERO` and `KEEP`, every promotion leaves one more deployment holding a slot, which blocks new pushes once you reach the limit. **To set the cleanup strategy**: ```bash Command theme={"system"} baseten model environment update-promotion --environment production \ --promotion-cleanup-strategy deactivate ``` For the full flag set, see [`model environment update-promotion`](/reference/cli/baseten/model-environment#update-promotion). **To set the cleanup strategy**: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar. 2. Select your model. 3. On the model's overview, open the environment card's **⋯** menu. 4. Choose **Configure promotion**. 5. For **After promotion**, select the action taken on the previous deployment. 6. Choose **Update** to apply the changes. **To set the cleanup strategy**: ```bash Request theme={"system"} curl -X PATCH "https://api.baseten.co/v1/models/{model_id}/environments/production" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "promotion_settings": { "promotion_cleanup_strategy": "DEACTIVATE" } }' ``` ```json Response theme={"system"} { "status": "ACCEPTED", "message": "Your request to update environment settings is complete" } ``` ## Next steps A rolling deployment starts from an ordinary promotion, so the same lifecycle operations drive it. * [Manage the deployment lifecycle](/deployment/manage/lifecycle) to run the promotion that starts a rolling deployment. * [Environments](/deployment/environments) for the concepts behind promotion settings. * [Scale a deployment](/deployment/manage/scaling) for the autoscaling settings a rollout respects. # Binary I/O Source: https://docs.baseten.co/development/chain/binaryio Performant serialization of numeric data Numeric data or audio/video are most efficiently transmitted as bytes. Other representations such as JSON or base64 encoding lose precision, add significant parsing overhead and increase message sizes (for example, \~33% increase for base64 encoding). Chains extends the JSON-centred pydantic ecosystem with two ways how you can include binary data: numpy array support and raw bytes. ## Numpy `ndarray` support Once you have your data represented as a numpy array, you can (and often without copying) convert it to `torch`, `tensorflow`, or other common numeric libraries' objects. To include numpy arrays in a pydantic model, chains has a special field type implementation `NumpyArrayField`. For example: ```python data_model.py theme={"system"} import numpy as np import pydantic from truss_chains import pydantic_numpy class DataModel(pydantic.BaseModel): some_numbers: pydantic_numpy.NumpyArrayField other_field: str ... numbers = np.random.random((3, 2)) data = DataModel(some_numbers=numbers, other_field="Example") print(data) # some_numbers=NumpyArrayField(shape=(3, 2), dtype=float64, data=[ # [0.39595027 0.23837526] # [0.56714894 0.61244946] # [0.45821942 0.42464844]]) # other_field='Example' ``` `NumpyArrayField` is a wrapper around the actual numpy array. Inside your python code, you can work with its `array` attribute: ```python theme={"system"} data.some_numbers.array += 10 # some_numbers=NumpyArrayField(shape=(3, 2), dtype=float64, data=[ # [10.39595027 10.23837526] # [10.56714894 10.61244946] # [10.45821942 10.42464844]]) # other_field='Example' ``` The interesting part is how it serializes when communicating between Chainlets or with a client. It can work in two modes: JSON and binary. ### Binary As a JSON alternative that supports byte data, Chains uses `msgpack` (with `msgpack_numpy`) to serialize the dict representation. For Chainlet-Chainlet RPCs this is done automatically for you by enabling binary mode of the dependency Chainlets, see [all options](/reference/sdk/chains#function-truss_chains-depends): ```python binary_rpc.py theme={"system"} import truss_chains as chains class Worker(chains.ChainletBase): async def run_remote(self, data: DataModel) -> DataModel: data.some_numbers.array += 10 return data class Consumer(chains.ChainletBase): def __init__(self, worker=chains.depends(Worker, use_binary=True)): self._worker = worker async def run_remote(self): numbers = np.random.random((3, 2)) data = DataModel(some_numbers=numbers, other_field="Example") result = await self._worker.run_remote(data) ``` Now the data is transmitted in a fast and compact way between Chainlets which often gives performance increases. ### Binary client If you want to send such data as input to a chain or parse binary output from a chain, you have to add the `msgpack` serialization client-side: ```python binary_client.py theme={"system"} import requests import msgpack import msgpack_numpy msgpack_numpy.patch() # Register hook for numpy. # Dump to "python" dict and then to binary. data_dict = data.model_dump(mode="python") data_bytes = msgpack.dumps(data_dict) # Set binary content type in request header. headers = { "Content-Type": "application/octet-stream", "Authorization": ... } response = requests.post(url, data=data_bytes, headers=headers) response_dict = msgpack.loads(response.content) response_model = ResponseModel.model_validate(response_dict) ``` The steps of dumping from a pydantic model and validating the response dict into a pydantic model can be skipped, if you prefer working with raw dicts on the client. The implementation of `NumpyArrayField` only needs `pydantic`, no other Chains dependencies. So you can take that implementation code in isolation and integrate it in your client code. Some version combinations of `msgpack` and `msgpack_numpy` give errors, we know that `msgpack = ">=1.0.2"` and `msgpack-numpy = ">=0.4.8"` work. ### JSON The JSON-schema to represent the array is a dict of `shape (tuple[int]), dtype (str), data_b64 (str)`. For example, ```python theme={"system"} print(data.model_dump_json()) '{"some_numbers":{"shape":[3,2],"dtype":"float64", "data_b64":"30d4/rnKJEAsvm...' ``` The base64 data corresponds to `np.ndarray.tobytes()`. To get back to the array from the JSON string, use the model's `model_validate_json` method. As discussed in the beginning, this schema is not performant for numeric data and only offered as a compatibility layer (JSON does not allow bytes); generally prefer the binary format. ## Simple `bytes` fields It is possible to add a `bytes` field to a pydantic model used in a chain, or as a plain argument to `run_remote`. This can be useful to include non-numpy data formats such as images or audio/video snippets. In this case, the "normal" JSON representation does not work and all involved requests or Chainlet-Chainlet-invocations must use binary mode. The same steps as for arrays [above](#binary-client) apply: construct dicts with `bytes` values and keys corresponding to the `run_remote` argument names or the field names in the pydantic model. Then use `msgpack` to serialize and deserialize those dicts. Don't forget to add `Content-type` headers and that `response.json()` will not work. # Concepts Source: https://docs.baseten.co/development/chain/concepts Glossary of Chains concepts and terminology This glossary defines the core Chains concepts you'll work with: Chainlets, their remote configuration and initialization, the `run_remote()` interface, entrypoints, and typed I/O. Read it alongside the [getting started guide](/development/chain/getting-started) when you build your first Chain. ## Chainlet A Chainlet is the basic building block of Chains. A Chainlet is a Python class that specifies: * A set of compute resources. * A Python environment with software dependencies. * A typed interface [`run_remote()`](/development/chain/concepts#run-remote-chaining-chainlets) for other Chainlets to call. This is the simplest possible Chainlet. Only the [`run_remote()`](/development/chain/concepts#run-remote-chaining-chainlets) method is required, and we can layer in other concepts to create a more capable Chainlet. ```python theme={"system"} import truss_chains as chains class SayHello(chains.ChainletBase): async def run_remote(self, name: str) -> str: return f"Hello, {name}" ``` You can modularize your code by creating your own chainlet sub-classes, refer to our [subclassing guide](/development/chain/subclassing). ### TrussChainlet `TrussChainlet` lets an existing Truss model, such as a custom server (vLLM) or a `model.py` implementation, join a Chain without being rewritten as a `ChainletBase` subclass. Set the `truss_dir` class attribute to the Truss directory: ```python theme={"system"} import truss_chains as chains class STT(chains.TrussChainlet): truss_dir = "./a_truss_model" ``` A `TrussChainlet` only receives calls: it can't be the chain's entrypoint and it can't call other chainlets. Other chainlets depend on it and call it through a `TrussHandle`. See [Add an existing Truss model to a Chain](/development/chain/truss-chainlets). ### Remote configuration Chainlets are meant for deployment as remote services. Each Chainlet specifies its own requirements for compute hardware (CPU count, GPU type and count, etc) and software dependencies (Python libraries or system packages). This configuration is built into a Docker image automatically as part of the deployment process. When no configuration is provided, the Chainlet will be deployed on a basic instance with one vCPU, 2GB of RAM, no GPU, and a standard set of Python and system packages. Configuration is set using the [`remote_config`](/reference/sdk/chains#remote-configuration) class variable within the Chainlet: ```python theme={"system"} import truss_chains as chains class MyChainlet(chains.ChainletBase): remote_config = chains.RemoteConfig( docker_image=chains.DockerImage( pip_requirements=["torch==2.3.0", ...] ), compute=chains.Compute(gpu="H100", ...), assets=chains.Assets(secret_keys=["hf_access_token"], ...), ) ``` To select an exact instance type instead of specifying individual resource fields, use `instance_type`: ```python theme={"system"} compute=chains.Compute(instance_type="H100:8x80") ``` When `instance_type` is specified, `cpu_count`, `memory`, and `gpu` fields are ignored. See the [remote configuration reference](/reference/sdk/chains#remote-configuration) for a complete list of options. ### Build commands Use `build_commands` to run shell commands during the Docker image build, after system packages are installed and before your Chainlet code is added. Useful for cloning repositories, pre-downloading model weights, or other setup work you want cached at build time so it does not run on every cold start. ```python theme={"system"} import truss_chains as chains class ComfyChainlet(chains.ChainletBase): remote_config = chains.RemoteConfig( compute=chains.Compute(gpu="A100"), build_commands=[ "git clone https://github.com/comfyanonymous/ComfyUI.git", "cd ComfyUI && pip install -r requirements.txt", ], ) ``` Each entry runs as a separate shell command in the order listed. This is the Chains equivalent of the Truss [`build_commands`](/development/model/dependencies#build-commands) field in `config.yaml`. ### Initialization Chainlets are implemented as classes because we often want to set up expensive static resources once at startup and then re-use it with each invocation of the Chainlet. For example, we only want to initialize an AI model and download its weights once then re-use it every time we run inference. We do this setup in `__init__()`, which is run exactly once when the Chainlet is deployed or scaled up. ```python theme={"system"} import truss_chains as chains class PhiLLM(chains.ChainletBase): def __init__(self) -> None: import torch import transformers self._model = transformers.AutoModelForCausalLM.from_pretrained( PHI_HF_MODEL, torch_dtype=torch.float16, device_map="auto", ) self._tokenizer = transformers.AutoTokenizer.from_pretrained( PHI_HF_MODEL, ) ``` Chainlet initialization also has two important features: context and dependency injection of other Chainlets, explained below. #### Context (access information) You can add a [`DeploymentContext`](/reference/sdk/chains#class-truss_chains-deploymentcontext) object as an optional argument to the `__init__`-method of a Chainlet. This allows you to use secrets within your Chainlet, such as using a `hf_access_token` to access a gated model on Hugging Face (note that when using secrets, they also need to be added to the `assets`). ```python theme={"system"} import truss_chains as chains class MistralLLM(chains.ChainletBase): remote_config = chains.RemoteConfig( ... assets = chains.Assets(secret_keys=["hf_access_token"], ...), ) def __init__( self, # Adding the `context` argument, allows us to access secrets context: chains.DeploymentContext = chains.depends_context(), ) -> None: import transformers # Using the secret from context to access a gated model on HF self._model = transformers.AutoModelForCausalLM.from_pretrained( "mistralai/Mistral-7B-Instruct-v0.2", use_auth_token=context.secrets["hf_access_token"], ) ``` #### Depends (call other Chainlets) The Chains framework uses the [`chains.depends()`](/reference/sdk/chains#function-truss_chains-depends) function in Chainlets' `__init__()` method to track the dependency relationship between different Chainlets within a Chain. This syntax, inspired by dependency injection, is used to translate local Python function calls into calls to the remote Chainlets in production. Once a dependency Chainlet is added with [`chains.depends()`](/reference/sdk/chains#function-truss_chains-depends), its [`run_remote()`](/development/chain/concepts#run-remote-chaining-chainlets) method can call this dependency Chainlet, for example, below `HelloAll` we can make calls to `SayHello`: ```python theme={"system"} import truss_chains as chains class HelloAll(chains.ChainletBase): def __init__(self, say_hello_chainlet=chains.depends(SayHello)) -> None: self._say_hello = say_hello_chainlet async def run_remote(self, names: list[str]) -> str: output = [] for name in names: output.append(self._say_hello.run_remote(name)) return "\n".join(output) ``` ## Run remote (chaining Chainlets) The `run_remote()` method is run each time the Chainlet is called. It is the sole public interface for the Chainlet (though you can have as many private helper functions as you want) and its inputs and outputs must have type annotations. In `run_remote()` you implement the actual work of the Chainlet, such as model inference or data chunking: ```python theme={"system"} import truss_chains as chains class PhiLLM(chains.ChainletBase): async def run_remote(self, messages: Messages) -> str: import torch model_inputs = await self._tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True ) inputs = await self._tokenizer(model_inputs, return_tensors="pt") input_ids = inputs["input_ids"].to("cuda") with torch.no_grad(): outputs = await self._model.generate( input_ids=input_ids, **self._generate_args) output_text = await self._tokenizer.decode( outputs[0], skip_special_tokens=True) return output_text ``` We recommend implementing this as an `async` method and using async APIs for doing all the work (for example, downloads, vLLM or TRT inference). It is possible to stream results back, see our [streaming guide](/development/chain/streaming). If `run_remote()` makes calls to other Chainlets, for example, invoking a dependency Chainlet for each element in a list, you can benefit from concurrent execution, by making the `run_remote()` an `async` method and starting the calls as concurrent tasks `asyncio.create_task(self._dep_chainlet.run_remote(...))`. ## Entrypoint The entrypoint is called directly from the deployed Chain's API endpoint and kicks off the entire chain. The entrypoint is also responsible for returning the final result back to the client. Using the [`@chains.mark_entrypoint`](/reference/sdk/chains#function-truss_chains-mark_entrypoint) decorator, one Chainlet within a file is set as the entrypoint to the chain. ```python theme={"system"} @chains.mark_entrypoint class HelloAll(chains.ChainletBase): ``` Optionally you can also set a Chain display name (not to be confused with Chainlet display name) with this decorator: ```python theme={"system"} @chains.mark_entrypoint("My Awesome Chain") class HelloAll(chains.ChainletBase): ``` ## I/O and `pydantic` data types To make orchestrating multiple remotely deployed services possible, Chains relies heavily on typed inputs and outputs. Values must be serialized to a safe exchange format to be sent over the network. The Chains framework uses the type annotations to infer how data should be serialized and currently is restricted to types that are JSON compatible. Types can be: * Direct type annotations for simple types such as `int`, `float`, or `list[str]`. * Pydantic models to define a schema for nested data structures or multiple arguments. An example of pydantic input and output types for a Chainlet is given below: ```python theme={"system"} import enum import pydantic class Modes(enum.Enum): MODE_0 = "MODE_0" MODE_1 = "MODE_1" class SplitTextInput(pydantic.BaseModel): data: str num_partitions: int mode: Modes class SplitTextOutput(pydantic.BaseModel): parts: list[str] part_lens: list[int] ``` Refer to the [pydantic docs](https://docs.pydantic.dev/latest/) for more details on how to define custom pydantic data models. Also refer to the [guide](/development/chain/binaryio) about efficient integration of binary and numeric data. ## Chains compared to Truss Chains is an alternate SDK for packaging and deploying AI models. It carries over many features and concepts from Truss and gives you access to the benefits of Baseten (resource provisioning, autoscaling, fast cold starts, etc), but it is not a 1-1 replacement for Truss. Here are some key differences: * Rather than running `truss init` and creating a Truss in a directory, a Chain is a single file, giving you more flexibility for implementing multi-step model inference. Create an example with `truss chains init`. * Configuration is done inline in typed Python code rather than in a `config.yaml` file. * While Chainlets are converted to Truss models when run on Baseten, `Chainlet != TrussModel`. Chains is designed for compatibility and incremental adoption, with a stub function for wrapping existing deployed models. You can also use `TrussChainlet` to integrate existing Truss directories directly into a Chain without rewriting them as `ChainletBase` subclasses. # Deploy Source: https://docs.baseten.co/development/chain/deploy Deploy your Chain on Baseten Deploying a Chain is an atomic action that deploys every Chainlet within the Chain. Each Chainlet specifies its own remote environment: hardware resources, Python and system dependencies, autoscaling settings. ## Published deployment By default, pushing a Chain creates a published deployment: ```sh Terminal theme={"system"} truss chains push ./my_chain.py ``` Where `my_chain.py` contains the entrypoint Chainlet for your Chain. Published deployments have access to full autoscaling settings. Each time you push, a new deployment is created. ## Development To create a development deployment for rapid iteration, use `--watch`: ```sh Terminal theme={"system"} truss chains push ./my_chain.py --watch ``` Development deployments are intended for testing and can't scale past one replica. Each time you make a development deployment, it overwrites the existing development deployment. Development deployments support rapid iteration with live code patching. See the [watch guide](/development/chain/watch). ## Environments To deploy a Chain to an environment, run: ```sh Terminal theme={"system"} truss chains push ./my_chain.py --environment {env_name} ``` Environments are intended for live traffic and have access to full autoscaling settings. Each time you deploy to an environment, a new deployment is created. Once the new deployment is live, it replaces the previous deployment, which is relegated to the published deployments list. [Learn more](/deployment/environments) about environments. # Architecture and design Source: https://docs.baseten.co/development/chain/design How to structure your Chainlets A Chain is composed of multiple connected Chainlets working together to perform a task. For example, the Chain in the following diagram takes a large audio file as input. Then it splits it into smaller chunks, transcribes each chunk in parallel (reducing the end-to-end latency), and finally aggregates and returns the results. To build an efficient Chain, we recommend drafting your high level structure as a flowchart or diagram. This can help you identify parallelizable units of work and steps that need different (model/hardware) resources. If one Chainlet creates many "sub-tasks" by calling other dependency Chainlets (for example, in a loop over partial work items), these calls should be done as `asyncio`-tasks that run concurrently. That way you get the most out of the parallelism that Chains offers. This design pattern is extensively used in the [audio transcription example](/examples/chains-audio-transcription). While using `asyncio` is essential for performance, it can also be tricky. Here are a few caveats to look out for: * Executing operations in an async function that block the event loop for more than a fraction of a second. This hinders the "flow" of processing requests concurrently and starting RPCs to other Chainlets. Ideally use native async APIs. Frameworks like vLLM or triton server offer such APIs, similarly file downloads can be made async and you might find [`AsyncBatcher`](https://github.com/hussein-awala/async-batcher) useful. If there is no async support, consider running blocking code in a thread/process pool (as an attribute of a Chainlet). * Creating async tasks (for example, with `asyncio.create_task`) does not start the task *immediately*. In particular, when starting several tasks in a loop, `create_task` must be alternated with operations that yield to the event loop that, so the task can be started. If the loop is not `async for` or contains other `await` statements, a "dummy" await can be added, for example `await asyncio.sleep(0)`. This allows the tasks to be started concurrently. # Engine Builder LLM models Source: https://docs.baseten.co/development/chain/engine-builder-models Engine-Builder LLM models are pre-trained models that are optimized for specific inference tasks. Baseten's [Engine-Builder](/engines/engine-builder-llm/overview) enables the deployment of optimized model inference engines. Currently, it supports TensorRT-LLM. Truss Chains lets you use these engines as Chainlets. ## Llama 7B example Use the `EngineBuilderLLMChainlet` baseclass to configure an LLM engine. The additional `engine_builder_config` field specifies model architecture, repository, engine parameters, and more; the full options are detailed in the [Engine-Builder configuration guide](/engines/engine-builder-llm/engine-builder-config). Define the engine-backed Chainlet: ```python llama_7b_chainlet.py theme={"system"} import truss_chains as chains from truss.base import trt_llm_config, truss_config class Llama7BChainlet(chains.EngineBuilderLLMChainlet): remote_config = chains.RemoteConfig( compute=chains.Compute(gpu=truss_config.Accelerator.H100), assets=chains.Assets(secret_keys=["hf_access_token"]), ) engine_builder_config = truss_config.TRTLLMConfiguration( build=trt_llm_config.TrussTRTLLMBuildConfiguration( base_model=trt_llm_config.TrussTRTLLMModel.LLAMA, checkpoint_repository=trt_llm_config.CheckpointRepository( source=trt_llm_config.CheckpointSource.HF, repo="meta-llama/Llama-3.1-8B-Instruct", ), max_batch_size=8, max_seq_len=4096, tensor_parallel_count=1, ) ) ``` ## Differences from standard Chainlets * No `run_remote` implementation: Unlike regular Chainlets, `EngineBuilderLLMChainlet` doesn't require users to implement `run_remote()`. Instead, it automatically wires into the deployed engine's API. All LLM Chainlets have the same function signature: `chains.EngineBuilderLLMInput` as input and a stream (`AsyncIterator`) of strings as output. Likewise, `EngineBuilderLLMChainlet`s can only be used as dependencies, but can't have dependencies themselves. * No `run_local` ([guide](/development/chain/localdev)) or `watch` ([guide](/development/chain/watch)). Standard Chains support a local debugging mode and watch; however, when using `EngineBuilderLLMChainlet`, local execution isn't available, and testing must be done after deployment. For a faster dev loop of the rest of your chain (everything except the engine-builder Chainlet), you can substitute those Chainlets with stubs, as you can for an already-deployed Truss model ([guide](/development/chain/stub)). ## Integrate the Engine-Builder chainlet After defining an `EngineBuilderLLMChainlet` like `Llama7BChainlet` above, you can use it as a dependency in other conventional Chainlets: ```python controller.py theme={"system"} from typing import AsyncIterator import truss_chains as chains @chains.mark_entrypoint class TestController(chains.ChainletBase): """Example using the Engine-Builder Chainlet in another Chainlet.""" def __init__(self, llm=chains.depends(Llama7BChainlet)) -> None: self._llm = llm async def run_remote(self, prompt: str) -> AsyncIterator[str]: messages = [{"role": "user", "content": prompt}] llm_input = chains.EngineBuilderLLMInput(messages=messages) async for chunk in self._llm.run_remote(llm_input): yield chunk ``` # Error handling Source: https://docs.baseten.co/development/chain/errorhandling Understanding and handling Chains errors Error handling in Chains follows the principle that the root cause bubbles up to the entrypoint, which returns an error response. This works like Python stack traces, which contain all the layers from where an exception was raised up to the main function. Consider the case of a Chain where the entrypoint calls `run_remote` of a Chainlet named `TextToNum` and this in turn invokes `TextReplicator`. The respective `run_remote` methods might also use other helper functions that appear in the call stack. Below is an example stack trace that shows how the root cause (a `ValueError`) is propagated up to the entrypoint's `run_remote` method (this is what you would see as an error log): ```text theme={"system"} Chainlet-Traceback (most recent call last): File "/packages/itest_chain.py", line 132, in run_remote value = self._accumulate_parts(text_parts.parts) File "/packages/itest_chain.py", line 144, in _accumulate_parts value += self._text_to_num.run_remote(part) ValueError: (showing chained remote errors, root error at the bottom) ├─ Error in dependency Chainlet `TextToNum`: │ Chainlet-Traceback (most recent call last): │ File "/packages/itest_chain.py", line 87, in run_remote │ generated_text = self._replicator.run_remote(data) │ ValueError: (showing chained remote errors, root error at the bottom) │ ├─ Error in dependency Chainlet `TextReplicator`: │ │ Chainlet-Traceback (most recent call last): │ │ File "/packages/itest_chain.py", line 52, in run_remote │ │ validate_data(data) │ │ File "/packages/itest_chain.py", line 36, in validate_data │ │ raise ValueError(f"This input is too long: {len(data)}.") ╰ ╰ ValueError: This input is too long: 100. ``` ## Exception handling and retries The stack trace above is what you see if you don't catch the exception. It is possible to add error handling around each remote Chainlet invocation. Chains tries to raise the same exception class on the *caller* Chainlet as was raised in the *dependency* Chainlet. * Builtin exceptions (for example, `ValueError`) always work. * Custom or third-party exceptions (for example, from `torch`) can be only raised in the caller if they are included in the dependencies of the caller as well. If the exception class cannot be resolved, a `GenericRemoteException` is raised instead. The *message* of re-raised exceptions is the concatenation of the original message and the formatted stack trace of the dependency Chainlet. Retry a remote invocation when it fails for transient reasons such as networking. Configure retries with `depends` [options](/reference/sdk/chains#function-truss_chains-depends). Below example shows how you can add automatic retries and error handling for the call to `TextReplicator` in `TextToNum`: ```python text_to_num.py theme={"system"} import truss_chains as chains class TextToNum(chains.ChainletBase): def __init__( self, replicator: TextReplicator = chains.depends(TextReplicator, retries=3), ) -> None: self._replicator = replicator async def run_remote(self, data: ...): try: generated_text = await self._replicator.run_remote(data) except ValueError: ... # Handle error. ``` ## Stack filtering The stack trace is intended to show the user implemented code in `run_remote` (and user implemented helper functions). Under the hood, the calls from one Chainlet to another go through an HTTP connection, managed by the Chains framework. And each Chainlet itself is run as a FastAPI server with several layers of request handling code "above". To provide concise, readable stacks, all of this non-user code is filtered out. # Your first Chain Source: https://docs.baseten.co/development/chain/getting-started Build and deploy two example Chains This quickstart guide contains instructions for creating two Chains: 1. A simple CPU-only "hello world"-Chain. 2. A Chain that implements Phi-3 Mini and uses it to write poems. ## Prerequisites You need [uv](https://docs.astral.sh/uv/) installed and a [Baseten account](https://app.baseten.co/signup) with an [API key](https://app.baseten.co/settings/account/api_keys). ## Hello World Chains are written in Python files. In your working directory, create `hello_chain/hello.py`: ```sh Terminal theme={"system"} mkdir hello_chain cd hello_chain touch hello.py ``` In the file, we'll specify a basic Chain. It has two Chainlets: * `HelloWorld`, the entrypoint, which handles the input and output. * `RandInt`, which generates a random integer. It is used a as a dependency by `HelloWorld`. Through the entrypoint, the Chain takes a maximum value and returns the string "Hello World!" repeated a variable number of times. ```python hello.py theme={"system"} import random import truss_chains as chains class RandInt(chains.ChainletBase): async def run_remote(self, max_value: int) -> int: return random.randint(1, max_value) @chains.mark_entrypoint class HelloWorld(chains.ChainletBase): def __init__(self, rand_int=chains.depends(RandInt, retries=3)) -> None: self._rand_int = rand_int async def run_remote(self, max_value: int) -> str: num_repetitions = await self._rand_int.run_remote(max_value) return "Hello World! " * num_repetitions ``` ### The Chainlet class-contract Exactly one Chainlet must be marked as the entrypoint with the [`@chains.mark_entrypoint`](/reference/sdk/chains#function-truss_chains-mark_entrypoint) decorator. This Chainlet is responsible for handling public-facing input and output for the whole Chain in response to an API call. A Chainlet class has a single public method, [`run_remote()`](/development/chain/concepts#run-remote-chaining-chainlets), which is the API endpoint for the entrypoint Chainlet and the function that other Chainlets can use as a dependency. The [`run_remote()`](/development/chain/concepts#run-remote-chaining-chainlets) method must be fully type-annotated with primitive python types or [pydantic models](https://docs.pydantic.dev/latest/). Chainlets cannot be naively instantiated. The only correct usages are: 1. Make one Chainlet depend on another one through the [`chains.depends()`](/reference/sdk/chains#function-truss_chains-depends) directive as an `__init__`-argument as shown above for the `RandInt` Chainlet. 2. In the [local debugging mode](/development/chain/localdev#test-a-chain-locally). Beyond that, you can structure your code as you like, with private methods, imports from other files, and so forth. Keep in mind that Chainlets are intended for distributed, replicated, remote execution, so using global variables, global state, and certain Python features like importing modules dynamically at runtime should be avoided as they may not work as intended. ### Deploy your Chain to Baseten To deploy your Chain to Baseten, run: ```bash Terminal theme={"system"} truss chains push --watch hello.py ``` The deploy command results in an output like this: ```text Output theme={"system"} ⛓️ HelloWorld - Chainlets ⛓️ ╭──────────────────────┬─────────────────────────┬─────────────╮ │ Status │ Name │ Logs URL │ ├──────────────────────┼─────────────────────────┼─────────────┤ │ 💚 ACTIVE │ HelloWorld (entrypoint) │ https://... │ ├──────────────────────┼─────────────────────────┼─────────────┤ │ 💚 ACTIVE │ RandInt (dep) │ https://... │ ╰──────────────────────┴─────────────────────────┴─────────────╯ Deployment succeeded. You can run the chain with: curl -X POST 'https://chain-.../run_remote' \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '' ``` Wait for the status to turn to `ACTIVE` and test invoking your Chain (replace `$INVOCATION_URL` in below command): ```bash Request theme={"system"} curl -X POST $INVOCATION_URL \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{"max_value": 10}' # "Hello World! Hello World! Hello World! " ``` ## Poetry with LLMs Our second example also has two Chainlets, but is somewhat more complex and realistic. The Chainlets are: * `PoemGenerator`, the entrypoint, which handles the input and output and orchestrates calls to the LLM. * `PhiLLM`, which runs inference on Phi-3 Mini. This Chain takes a list of words and returns a poem about each word, written by Phi-3. Here's the architecture: We build this Chain in a new working directory (if you are still inside `hello_chain/`, go up one level with `cd ..` first): ```sh Terminal theme={"system"} mkdir poetry_chain cd poetry_chain touch poems.py ``` A similar end-to-end code example, using Mistral as an LLM, is available in the [examples repo](https://github.com/basetenlabs/model/tree/main/truss-chains/examples/mistral). ### Build the LLM Chainlet The main difference between this Chain and the previous one is that we now have an LLM that needs a GPU and more complex dependencies. Copy the following code into `poems.py`: ```python poems.py theme={"system"} import asyncio from typing import List import pydantic import truss_chains as chains from truss import truss_config PHI_HF_MODEL = "microsoft/Phi-3-mini-4k-instruct" PHI_MOUNT = "/models/phi" # Mount the model weights from Hugging Face into the Chainlet container at runtime. PHI_WEIGHTS = truss_config.WeightsSource( source=f"hf://{PHI_HF_MODEL}@main", mount_location=PHI_MOUNT, allow_patterns=["*.json", "*.safetensors", ".model"], ) class Messages(pydantic.BaseModel): messages: List[dict[str, str]] class PhiLLM(chains.ChainletBase): # `remote_config` defines the resources required for this chainlet. remote_config = chains.RemoteConfig( docker_image=chains.DockerImage( # The phi model needs some extra python packages. pip_requirements=[ "accelerate==0.30.1", "einops==0.8.0", "transformers==4.41.2", "torch==2.3.0", ] ), # The phi model needs a GPU and more CPUs. compute=chains.Compute(cpu_count=2, gpu="T4"), # Mount the model weights at runtime through BDN. assets=chains.Assets(weights=[PHI_WEIGHTS]), ) def __init__(self) -> None: # Note the imports of the *specific* python requirements are # pushed down to here. This code will only be executed on the # remotely deployed Chainlet, not in the local environment, # so we don't need to install these packages in the local # dev environment. import torch import transformers self._model = transformers.AutoModelForCausalLM.from_pretrained( PHI_MOUNT, torch_dtype=torch.float16, device_map="auto", ) self._tokenizer = transformers.AutoTokenizer.from_pretrained( PHI_MOUNT, ) self._generate_args = { "max_new_tokens" : 512, "temperature" : 1.0, "top_p" : 0.95, "top_k" : 50, "repetition_penalty" : 1.0, "no_repeat_ngram_size": 0, "use_cache" : True, "do_sample" : True, "eos_token_id" : self._tokenizer.eos_token_id, "pad_token_id" : self._tokenizer.pad_token_id, } async def run_remote(self, messages: Messages) -> str: import torch model_inputs = self._tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True ) inputs = self._tokenizer(model_inputs, return_tensors="pt") input_ids = inputs["input_ids"].to("cuda") with torch.no_grad(): outputs = self._model.generate( input_ids=input_ids, **self._generate_args) output_text = self._tokenizer.decode( outputs[0], skip_special_tokens=True) return output_text ``` ### Build the entrypoint Now that we have an LLM, we can use it in a poem generator Chainlet. Add the following code to `poems.py`: ```python poems.py theme={"system"} import asyncio @chains.mark_entrypoint class PoemGenerator(chains.ChainletBase): def __init__(self, phi_llm: PhiLLM = chains.depends(PhiLLM)) -> None: self._phi_llm = phi_llm async def run_remote(self, words: list[str]) -> list[str]: tasks = [] for word in words: messages = Messages( messages=[ { "role" : "system", "content": ( "You are poet who writes short, " "lighthearted, amusing poetry." ), }, {"role": "user", "content": f"Write a poem about {word}"}, ] ) tasks.append( asyncio.create_task(self._phi_llm.run_remote(messages))) await asyncio.sleep(0) # Yield to event loop, to allow starting tasks. return list(await asyncio.gather(*tasks)) ``` We use `asyncio.create_task` around each RPC to the LLM chainlet. This makes the current python process start these remote calls concurrently, that is, the next call is started before the previous one has finished and we can minimize our overall runtime. To await the results of all calls, `asyncio.gather` is used which gives us back normal python objects. If the LLM is hit with many concurrent requests, it can auto-scale up (if autoscaling is configured). More advanced LLM models have batching capabilities, so for those even a single instance can serve concurrent request. ### Deploy your Chain to Baseten To deploy your Chain to Baseten, run: ```bash Terminal theme={"system"} truss chains push --watch poems.py ``` Wait for the status to turn to `ACTIVE` and test invoking your Chain (replace `$INVOCATION_URL` in below command): ```bash Request theme={"system"} curl -X POST $INVOCATION_URL \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{"words": ["bird", "plane", "superman"]}' #[[ #" [INST] Generate a poem about: bird [/INST] In the quiet hush of...", #" [INST] Generate a poem about: plane [/INST] In the vast, boundless...", #" [INST] Generate a poem about: superman [/INST] In the realm where..." #]] ``` # Invocation Source: https://docs.baseten.co/development/chain/invocation Call your deployed Chain Once your Chain is deployed, you can call it through its API endpoint. Chains use the same inference API as models: * [Environment endpoint](/reference/inference-api/predict-endpoints/environments-run-remote) * [Development endpoint](/reference/inference-api/predict-endpoints/development-run-remote) * [Endpoint by ID](/reference/inference-api/predict-endpoints/deployment-run-remote) Here's an example which calls the development deployment: ```python call_chain.py theme={"system"} import requests import os # From the Chain overview page on Baseten # E.g. "https://chain-.api.baseten.co/development/run_remote" CHAIN_URL = "" baseten_api_key = os.environ["BASETEN_API_KEY"] # JSON keys and types match the `run_remote` method signature. data = {...} resp = requests.post( CHAIN_URL, headers={"Authorization": f"Bearer {baseten_api_key}"}, json=data, ) print(resp.json()) ``` ## How to pass chain input The data schema of the inference request corresponds to the function signature of [`run_remote()`](/development/chain/concepts#run-remote-chaining-chainlets) in your entrypoint Chainlet. For example, for the Hello Chain, `HelloAll.run_remote()`: ```python theme={"system"} async def run_remote(self, names: list[str]) -> str: ``` You'd pass the following JSON payload: ```json Payload theme={"system"} { "names": ["Marius", "Sid", "Bola"] } ``` That is, the keys in the JSON record match the argument names, and values match the types of `run_remote`. ## Async chain inference Like Truss models, Chains support async invocation. The [guide for models](/inference/async) applies largely. In particular for how to wrap the input and set up the webhook to process results. The following additional points are chains specific: * Use chain-based URLS: * `https://chain-{chain}.api.baseten.co/production/async_run_remote` * `https://chain-{chain}.api.baseten.co/development/async_run_remote` * `https://chain-{chain}.api.baseten.co/deployment/{deployment}/async_run_remote`. * `https://chain-{chain}.api.baseten.co/environments/{env_name}/async_run_remote`. * Only the entrypoint is invoked asynchronously. Internal Chainlet-Chainlet calls run synchronously. # Local development Source: https://docs.baseten.co/development/chain/localdev Iterating, Debugging, Testing, Mocking Chains run in production as replicated remote deployments, but you can develop and test them locally first. Chains exists to help you build multi-step, multi-model pipelines. The abstractions that Chains introduces are based on six opinionated principles: three for architecture and three for developer experience. **Architecture principles** Each step in the pipeline can set its own hardware requirements and software dependencies, separating GPU and CPU workloads. Each component has independent autoscaling parameters for targeted resource allocation, removing bottlenecks from your pipelines. Components specify a single public interface for flexible-but-safe composition and are reusable between projects **Developer experience principles** Eliminate entire taxonomies of bugs by writing typed Python code and validating inputs, outputs, module initializations, function signatures, and even remote server configurations. Seamless local testing and cloud deployments: test Chains locally with support for mocking the output of any step and simplify your cloud deployment loops by separating large model deployments from quick updates to glue code. Use Chains to orchestrate existing model deployments, like pre-packaged models from Baseten’s model library, alongside new model pipelines built entirely within Chains. Locally, a Chain is just Python files in a source tree. While that gives you a lot of flexibility in how you structure your code, there are some constraints and rules to follow to ensure successful distributed, remote execution in production. The best thing you can do while developing locally with Chains is to run your code frequently, even if you do not have a `__main__` section: the Chains framework runs various validations at module initialization to help you catch issues early. Additionally, running `mypy` and fixing reported type errors can help you find problems early in a rapid feedback loop, before attempting a (much slower) deployment. Complementary to the purely local development Chains also has a "watch" mode, like Truss, see the [watch guide](/development/chain/watch). ## Test a Chain locally Let's revisit our "Hello World" Chain: ```python hello_chain/hello.py theme={"system"} import asyncio import truss_chains as chains # This Chainlet does the work class SayHello(chains.ChainletBase): async def run_remote(self, name: str) -> str: return f"Hello, {name}" # This Chainlet orchestrates the work @chains.mark_entrypoint class HelloAll(chains.ChainletBase): def __init__(self, say_hello_chainlet=chains.depends(SayHello)) -> None: self._say_hello = say_hello_chainlet async def run_remote(self, names: list[str]) -> str: tasks = [] for name in names: tasks.append(asyncio.create_task( self._say_hello.run_remote(name))) return "\n".join(await asyncio.gather(*tasks)) # Test the Chain locally if __name__ == "__main__": with chains.run_local(): hello_chain = HelloAll() result = asyncio.run(hello_chain.run_remote(["Marius", "Sid", "Bola"])) print(result) ``` When the `__main__()` module is run, local instances of the Chainlets are created, allowing you to test functionality of your chain just by executing the Python file: ```bash Terminal theme={"system"} cd hello_chain python hello.py # Hello, Marius # Hello, Sid # Hello, Bola ``` ## Mock execution of GPU Chainlets Using `run_local()` to run your code locally requires that your development environment have the compute resources and dependencies that each Chainlet needs. But that often isn't possible when building with AI models. Chains offers a workaround, mocking, to let you test the coordination and business logic of your multi-step inference pipeline without worrying about running the model locally. The second example in the [getting started guide](/development/chain/getting-started) implements a Truss Chain for generating poems with Phi-3. This Chain has two Chainlets: 1. The `PhiLLM` Chainlet, which can run on NVIDIA GPUs such as the L4. 2. The `PoemGenerator` Chainlet, which easily runs on a CPU. If you have an NVIDIA T4 under your desk, good for you. For the rest of us, we can mock the `PhiLLM` Chainlet that is infeasible to run locally so that we can quickly test the `PoemGenerator` Chainlet. To do this, we define a mock Phi-3 model in our `__main__` module and give it a [`run_remote()`](/development/chain/concepts#run-remote-chaining-chainlets) method that produces a test output that matches the output type we expect from the real Chainlet. Then, we inject an instance of this mock Chainlet into our Chain: ```python poems.py theme={"system"} if __name__ == "__main__": class FakePhiLLM: async def run_remote(self, prompt: str) -> str: return f"Here's a poem about {prompt.split(' ')[-1]}" with chains.run_local(): poem_generator = PoemGenerator(phi_llm=FakePhiLLM()) result = asyncio.run(poem_generator.run_remote(words=["bird", "plane", "superman"])) print(result) ``` And run your Python file: ```bash Terminal theme={"system"} python poems.py # ['Here's a poem about bird', 'Here's a poem about plane', 'Here's a poem about superman'] ``` ### Typing of mocks You may notice that the argument `phi_llm` expects a type `PhiLLM`, while we pass an instance of `FakePhiLLM`. These aren't the same, which is formally a type error. However, this works at runtime because we constructed `FakePhiLLM` to implement the same *protocol* as the real thing. We can make this explicit by defining a `Protocol` as a type annotation: ```python theme={"system"} from typing import Protocol class PhiProtocol(Protocol): def run_remote(self, data: str) -> str: ... ``` and changing the argument type in `PoemGenerator`: ```python theme={"system"} @chains.mark_entrypoint class PoemGenerator(chains.ChainletBase): def __init__(self, phi_llm: PhiProtocol = chains.depends(PhiLLM)) -> None: self._phi_llm = phi_llm ``` The `Protocol` annotation is optional; it makes the typing consistency explicit. # Overview Source: https://docs.baseten.co/development/chain/overview Chains is a framework for building robust, performant multi-step and multi-model inference pipelines and deploying them to production. It addresses the common challenges of managing latency, cost and dependencies for complex workflows, while leveraging Truss' existing battle-tested performance, reliability and developer toolkit.