# Baseten > This file is very large and may be truncated. For an index of this content, please refer to https://docs.baseten.co/llms.txt instead. # Manage Baseten with an agent Source: https://docs.baseten.co/agent-setup Install the Baseten skill and MCP servers so your agent can manage your Baseten workspace and search these docs. Install the Baseten skill and MCP servers so your coding agent can manage your Baseten workspace and search these docs. To connect Claude Code, Codex CLI, Pi, Droid, or OpenCode to Model APIs, see [Use models in coding agents](/inference/model-apis/coding-agents). If your organization uses Baseten Grid, see [Connect your coding agent to Grid](/grid/developers/set-up). ## Install the agent toolkit > [tab] Prompt > > Paste this prompt into your coding agent: > > ```md theme={"system"} > Install the Baseten agent toolkit globally for every coding agent on this machine: > > 1. Install the `baseten` skill: `npx skills add basetenlabs/baseten-skills -g -y` > 2. Add the backend MCP server: `npx add-mcp https://api.baseten.co/mcp -g -y` > Don't add an Authorization header or API key. The server uses OAuth, and I'll sign in to Baseten in my browser when it first connects. > 3. Add the docs MCP server: `npx add-mcp https://docs.baseten.co/mcp -n baseten_docs -g -y` > > If `add-mcp` stops because Claude Desktop doesn't support remote servers, rerun both `add-mcp` commands with an `-a` flag for each coding agent I use, such as `-a claude-code -a cursor`. > > Then tell me how to sign in to the backend server, how to verify the setup, and whether I need to restart my agent. > ``` > [tab] Commands > > **To install the skill and MCP servers**: > > Run these commands. The `-g` flag makes the installation available to each detected agent, and `-y` skips confirmation prompts: > > ```bash theme={"system"} > npx skills add basetenlabs/baseten-skills -g -y > npx add-mcp https://api.baseten.co/mcp -g -y > npx add-mcp https://docs.baseten.co/mcp -n baseten_docs -g -y > ``` > > If Claude Desktop is installed, `add-mcp` stops without configuring any agent, because Claude Desktop accepts remote servers only as connectors. Name the agents you use with `-a` instead: > > ```bash theme={"system"} > npx add-mcp https://api.baseten.co/mcp -g -y -a claude-code -a cursor > npx add-mcp https://docs.baseten.co/mcp -n baseten_docs -g -y -a claude-code -a cursor > ``` > > Run `npx add-mcp list-agents` for the other agent names. Restart your coding agent so it loads the skill and MCP configuration. Then start the sign-in from your agent. It opens a browser window where you sign in to Baseten and approve access. After that, the agent refreshes its own access tokens. ## Install the Baseten CLI The skill deploys and operates models through the [Baseten CLI](/reference/cli/baseten/overview), which is built for agents: it never prompts when stdin isn't a terminal, `--wait` blocks until a deployment is active and exits non-zero on failure, and every command supports `--output json` and `--jq`. > [tab] Prompt > > Paste this prompt into your coding agent: > > ```md theme={"system"} > Install the Baseten CLI on this machine. > > First, run `baseten version` to check whether it's already installed. If it is, report the version. > > If it's missing, install it with Homebrew on macOS or Linux: > > brew tap basetenlabs/baseten > brew trust --formula basetenlabs/baseten/baseten > brew install baseten > > Then run `baseten version` to verify the installation and report the installed version. > > If Homebrew isn't available or I'm using Windows, report that these installation commands don't apply to my system. > ``` > [tab] Commands > > Install with Homebrew on macOS or Linux: > > ```sh theme={"system"} > brew tap basetenlabs/baseten > brew trust --formula basetenlabs/baseten/baseten > brew install baseten > ``` > > On Windows, or to pin a version, see the [Baseten CLI install reference](/reference/cli/baseten/overview#install). > [tab] Claude Code > > **To sign in and confirm that the servers are connected**: > > Run `/mcp`, select `baseten`, and choose **Authenticate** to sign in. Then run `/mcp` again. You should see: > > ```output theme={"system"} > baseten ✔ connected > baseten_docs ✔ connected > ``` > > If Claude Code prompts before reading skill files, add this to `~/.claude/settings.json`: > > ```json theme={"system"} > { > "permissions": { > "allow": ["Read(~/.claude/skills/**)"] > } > } > ``` > [tab] Cursor or VS Code > > **To sign in and confirm that the servers are connected**: > > Open MCP settings, sign in when `baseten` prompts you, and confirm both `baseten` and `baseten_docs` show as connected. Verify the setup by asking your agent to list the models in your workspace. If your agent exposes skills as slash commands, you can also run `/baseten`. The backend MCP server acts as you. It works in your account's workspace and has your account's permissions. ## Run a quickstart with your agent Set `BASETEN_API_KEY` to your [API key](/organization/api-keys) in your shell, then start your agent from that shell and paste a prompt: > [tab] Model APIs > > Run the [Model APIs quickstart](/quickstart): > > ```md theme={"system"} > Read https://docs.baseten.co/quickstart.md and walk me through its manual steps to call a Model API. Read my API key from BASETEN_API_KEY rather than asking me for it. Before you send a request, show it to me and explain how it bills. > ``` > [tab] Dedicated Inference > > Deploy a model with the [model server guide](/development/model/build-your-first-model): > > ```md theme={"system"} > Read https://docs.baseten.co/development/model/build-your-first-model.md and walk me through its steps to deploy a model. Read my API key from BASETEN_API_KEY rather than asking me for it. Before you deploy, show me the model, its GPU, and how the deployment bills, and wait for my go-ahead. At the end, tell me what is still running and check with me before you deactivate or delete it. > ``` > [tab] Training > > Run the [Loops quickstart](/loops/quickstart): > > ```md theme={"system"} > Read https://docs.baseten.co/loops/quickstart.md and walk me through its steps: train Qwen3.6-35B-A3B with reinforcement learning on serverless Loops, then sample from it. Read my API key from BASETEN_API_KEY rather than asking me for it. Show me each command and how it bills, and wait for my go-ahead before you start a run. At the end, list anything still running and check with me before you deactivate it. > ``` ## Configure your agent manually Use these instructions if the installer doesn't detect your agent. Add both MCP servers to your agent's configuration, then install the skill with `npx skills add basetenlabs/baseten-skills`. For a docs-only setup, add only `baseten_docs`. > [tab] Cursor > > **To add the MCP servers by hand**: > > Add this to `mcp.json`: > > ```json mcp.json theme={"system"} > { > "mcpServers": { > "baseten": { "type": "http", "url": "https://api.baseten.co/mcp" }, > "baseten_docs": { "type": "http", "url": "https://docs.baseten.co/mcp" } > } > } > ``` > [tab] VS Code > > **To add the MCP servers by hand**: > > Add this to `.vscode/mcp.json`: > > ```json .vscode/mcp.json theme={"system"} > { > "servers": { > "baseten": { "type": "http", "url": "https://api.baseten.co/mcp" }, > "baseten_docs": { "type": "http", "url": "https://docs.baseten.co/mcp" } > } > } > ``` > [tab] Claude Code > > **To add the MCP servers by hand**: > > 1. Install the skill: > ```bash theme={"system"} > npx skills add basetenlabs/baseten-skills > ``` > 2. Add the backend MCP server: > ```bash theme={"system"} > claude mcp add --transport http baseten https://api.baseten.co/mcp > ``` > 3. Add the docs MCP server: > ```bash theme={"system"} > claude mcp add --transport http baseten_docs https://docs.baseten.co/mcp > ``` `npx add-mcp` and `npx skills add` also detect Codex, Antigravity, Goose, Windsurf, and other supported agents. For GUI clients such as Claude Desktop, add both server URLs as custom connectors. Any MCP-compatible agent can use these URLs. You sign in to Baseten the first time each agent connects to the backend server. ## Connect with an API key Agents that run without a browser, such as agents in CI, can't complete the sign-in. Connect them with an API key that has management permissions instead. Create a dedicated key in your [API key settings](https://app.baseten.co/settings/api_keys) so you can revoke the agent's access without affecting inference or other tools. **To add the backend MCP server with an API key**: Store the key in the `BASETEN_MCP_KEY` environment variable, for example from your CI secret store. Then run the following command, which reads the key from the variable so it stays out of your shell history: ```bash theme={"system"} npx add-mcp https://api.baseten.co/mcp -g -y --header "Authorization: Bearer ${BASETEN_MCP_KEY}" ``` The server connects to the workspace that owns the key. ## Next steps > [card] [Use models in coding agents](/inference/model-apis/coding-agents) > > Call Model APIs from Pi, Droid, or OpenCode, or install Baseten Switch for Claude Code, Codex CLI, or Pi. > [card] [Deploy your first model](/development/model/build-your-first-model) > > Deploy a Hugging Face model with vLLM and the Baseten CLI. > [card] [Sandboxes](/sandboxes/overview) > > Give your application's agents isolated environments to build and run code. > [accordion] Pull docs into an agent without MCP > > Append `.md` to any docs URL to open the page as Markdown. You can also use [llms.txt](https://docs.baseten.co/llms.txt) to index the documentation or [llms-full.txt](https://docs.baseten.co/llms-full.txt) to retrieve its full text. The **Copy page** button in the upper-right corner of each page copies the Markdown or opens it as plain text: > > ![The page context menu with options to copy or open documentation as Markdown.](https://mintcdn.com/baseten-preview/f3Xm7Ysn9_rbmtpW/images/ai-tools-contextual-menu.png?fit=max\&auto=format\&n=f3Xm7Ysn9_rbmtpW\&q=85\&s=93400b5d90ac95c158618ad846555bc3) > > If you cannot install the full `baseten` skill, install the lightweight docs skill instead: > > ```bash theme={"system"} > npx skills add https://docs.baseten.co > ``` # Cancel a queued async request. Source: https://docs.baseten.co/api-reference/cancel-a-queued-async-request /reference/inference-api/inference-api-spec.json delete /async_request/{request_id} Cancels an async request. Only requests with `QUEUED` status may be canceled. Rate limited to 20 requests per second. # Get the status of an async request. Source: https://docs.baseten.co/api-reference/get-the-status-of-an-async-request /reference/inference-api/inference-api-spec.json get /async_request/{request_id} Returns the current status of an async model or chain request. Rate limited to 20 requests per second. # Asynchronously call a named environment of a chain. Source: https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-named-environment-of-a-chain /reference/inference-api/inference-api-spec.json post /environments/{env_name}/async_run_remote # Asynchronously call a named environment of a model. Source: https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-named-environment-of-a-model /reference/inference-api/inference-api-spec.json post /environments/{env_name}/async_predict # Asynchronously call a specific deployment of a chain. Source: https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-specific-deployment-of-a-chain /reference/inference-api/inference-api-spec.json post /deployment/{deployment_id}/async_run_remote # Asynchronously call a specific deployment of a model. Source: https://docs.baseten.co/api-reference/non-regional/asynchronously-call-a-specific-deployment-of-a-model /reference/inference-api/inference-api-spec.json post /deployment/{deployment_id}/async_predict # Asynchronously call the development deployment of a chain. Source: https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-development-deployment-of-a-chain /reference/inference-api/inference-api-spec.json post /development/async_run_remote # Asynchronously call the development deployment of a model. Source: https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-development-deployment-of-a-model /reference/inference-api/inference-api-spec.json post /development/async_predict # Asynchronously call the production environment of a chain. Source: https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-production-environment-of-a-chain /reference/inference-api/inference-api-spec.json post /production/async_run_remote Enqueues an asynchronous request for the chain deployment promoted to the production environment. # Asynchronously call the production environment of a model. Source: https://docs.baseten.co/api-reference/non-regional/asynchronously-call-the-production-environment-of-a-model /reference/inference-api/inference-api-spec.json post /production/async_predict Enqueues an asynchronous predict request for the deployment promoted to the production environment. Returns a request ID that can be used to poll for status or cancel the request. # Call a specific chain deployment by deployment ID. Source: https://docs.baseten.co/api-reference/non-regional/call-a-specific-chain-deployment-by-deployment-id /reference/inference-api/inference-api-spec.json post /deployment/{deployment_id}/run_remote # Call a specific deployment of a model by deployment ID. Source: https://docs.baseten.co/api-reference/non-regional/call-a-specific-deployment-of-a-model-by-deployment-id /reference/inference-api/inference-api-spec.json post /deployment/{deployment_id}/predict Sends a synchronous predict request to the specified deployment. # Call the chain deployment associated with a specified environment. Source: https://docs.baseten.co/api-reference/non-regional/call-the-chain-deployment-associated-with-a-specified-environment /reference/inference-api/inference-api-spec.json post /environments/{env_name}/run_remote # Call the development deployment of a chain. Source: https://docs.baseten.co/api-reference/non-regional/call-the-development-deployment-of-a-chain /reference/inference-api/inference-api-spec.json post /development/run_remote # Call the development deployment of a model. Source: https://docs.baseten.co/api-reference/non-regional/call-the-development-deployment-of-a-model /reference/inference-api/inference-api-spec.json post /development/predict Sends a synchronous predict request to the development deployment. # Call the model deployment associated with a specified environment. Source: https://docs.baseten.co/api-reference/non-regional/call-the-model-deployment-associated-with-a-specified-environment /reference/inference-api/inference-api-spec.json post /environments/{env_name}/predict Sends a synchronous predict request to the deployment promoted to the specified environment. # Call the production environment of a chain. Source: https://docs.baseten.co/api-reference/non-regional/call-the-production-environment-of-a-chain /reference/inference-api/inference-api-spec.json post /production/run_remote Sends a synchronous request to the chain deployment promoted to the production environment. The request body is forwarded to the chain's `run_remote` entrypoint. # Call the production environment of a model. Source: https://docs.baseten.co/api-reference/non-regional/call-the-production-environment-of-a-model /reference/inference-api/inference-api-spec.json post /production/predict Sends a synchronous predict request to the deployment promoted to the production environment. The request body is forwarded directly to the model's `predict` function. # Get async queue status for a named environment. Source: https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-a-named-environment /reference/inference-api/inference-api-spec.json get /environments/{env_name}/async_queue_status # Get async queue status for a specific deployment. Source: https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-a-specific-deployment /reference/inference-api/inference-api-spec.json get /deployment/{deployment_id}/async_queue_status # Get async queue status for the development deployment. Source: https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-the-development-deployment /reference/inference-api/inference-api-spec.json get /development/async_queue_status # Get async queue status for the production environment. Source: https://docs.baseten.co/api-reference/non-regional/get-async-queue-status-for-the-production-environment /reference/inference-api/inference-api-spec.json get /production/async_queue_status Returns the number of queued and in-progress async requests for the deployment promoted to the production environment. Rate limited to 20 requests per second. # Wake a named environment of a model. Source: https://docs.baseten.co/api-reference/non-regional/wake-a-named-environment-of-a-model /reference/inference-api/inference-api-spec.json post /environments/{env_name}/wake # Wake a specific deployment of a model by deployment ID. Source: https://docs.baseten.co/api-reference/non-regional/wake-a-specific-deployment-of-a-model-by-deployment-id /reference/inference-api/inference-api-spec.json post /deployment/{deployment_id}/wake # Wake the development deployment of a model. Source: https://docs.baseten.co/api-reference/non-regional/wake-the-development-deployment-of-a-model /reference/inference-api/inference-api-spec.json post /development/wake # Wake the production environment of a model. Source: https://docs.baseten.co/api-reference/non-regional/wake-the-production-environment-of-a-model /reference/inference-api/inference-api-spec.json post /production/wake Triggers a wake for the deployment promoted to the production environment. Returns immediately with 202 Accepted. # Asynchronously call a regional environment of a chain. Source: https://docs.baseten.co/api-reference/regional/asynchronously-call-a-regional-environment-of-a-chain /reference/inference-api/inference-api-spec.json post /async_run_remote Enqueues an asynchronous run_remote request via a regional hostname. The environment is determined by the hostname, not the path. # Asynchronously call a regional environment of a model. Source: https://docs.baseten.co/api-reference/regional/asynchronously-call-a-regional-environment-of-a-model /reference/inference-api/inference-api-spec.json post /async_predict Enqueues an asynchronous predict request via a regional hostname. The environment is determined by the hostname, not the path. # Call a regional environment of a chain. Source: https://docs.baseten.co/api-reference/regional/call-a-regional-environment-of-a-chain /reference/inference-api/inference-api-spec.json post /run_remote Sends a synchronous run_remote request via a regional hostname. The environment is determined by the hostname, not the path. # Call a regional environment of a model. Source: https://docs.baseten.co/api-reference/regional/call-a-regional-environment-of-a-model /reference/inference-api/inference-api-spec.json post /predict Sends a synchronous predict request via a regional hostname. The environment is determined by the hostname, not the path. # Get async queue status for a regional environment. Source: https://docs.baseten.co/api-reference/regional/get-async-queue-status-for-a-regional-environment /reference/inference-api/inference-api-spec.json get /async_queue_status # Wake a regional environment of a model. Source: https://docs.baseten.co/api-reference/regional/wake-a-regional-environment-of-a-model /reference/inference-api/inference-api-spec.json post /wake # How Baseten works Source: https://docs.baseten.co/concepts/howbasetenworks The moving parts behind training, deployment, request routing, autoscaling, and environment promotion on Baseten. You can deploy an existing model or train a new one and deploy its checkpoint. This page explains the systems behind both paths: build pipelines, request routing, autoscaling, cold starts, and environments. For product options and starting points, see [Baseten overview](/overview). ## Multi-cloud Capacity Management (MCM) Multi-cloud Capacity Management (MCM) provisions and manages GPUs across cloud providers and geographic regions. When you request hardware, such as an H100 in US-East-1 or a cluster of B200s in a private region, MCM provisions it, configures networking, and monitors its health. Baseten provides a consistent training and inference runtime across the underlying infrastructure. MCM also supports high availability. Deployments run active-active across clusters and clouds. If a region or provider loses capacity, MCM reroutes and reprovisions workloads. ## Deploy an existing model To deploy a model, describe the model in a `config.yaml` (for supported architectures) or a small Python `Model` class (for custom code), then run `baseten model push` to ship it. The `config.yaml` follows [Truss](https://pypi.org/project/truss/), Baseten's open-source model packaging format. **1. Upload project** `baseten model push` validates your `config.yaml`, archives your project directory, and uploads it to cloud storage. Baseten receives the archive and starts the build. **2. Process model weights** For [Engine-Builder-LLM](/engines/engine-builder-llm/overview), Baseten downloads model weights from the source repository (Hugging Face, S3, or GCS) and compiles them with TensorRT-LLM. Compilation builds optimized CUDA kernels for the target GPU architecture, applies quantization if configured, and sets up tensor parallelism across multiple GPUs. **3. Package and deploy** Baseten packages the compiled engine, runtime configuration, and serving infrastructure into a container, deploys it to GPU infrastructure, and exposes it as an API endpoint. `baseten model push` returns once the upload finishes. For engine-based deployments, compilation can take several minutes. Watch progress in the deployment logs, or wait for the dashboard to show "Active." For [custom model code](/development/model/model-class), Baseten installs your Python dependencies, packages the `Model` class into a container, and deploys it. Custom builds do not apply Baseten engine optimizations automatically. Each push produces a container image identified by a content hash and stored in Baseten's container registry. The image is immutable, and an unchanged project reuses the cached image instead of triggering a new build. ## Train a model Use [Loops](/loops/overview) to write a Python training loop that calls a dedicated trainer and sampler. Use [Training Jobs](/training/overview) to run your own training container to completion. To run a training job, define the job in a Python configuration file (typically `config.py`) using the [`truss_train` SDK](/reference/sdk/training), then submit it with `baseten train push --config config.py`. Baseten provisions GPUs through MCM, runs your training container, and syncs checkpoints to storage as the job progresses. **1. Submit the job** `baseten train push --config config.py` packages your training config, uploads it to Baseten, and starts the job on the hardware you specified (H100 or H200, single-node or multi-node). Your training code can use Axolotl, TRL, VeRL, Megatron, or any other framework you bundle into the container. **2. Run and checkpoint** Baseten runs your training container on the provisioned GPUs. As your training code writes checkpoints to the configured directory, Baseten uploads them to durable storage. If the job fails or you stop it, you can still use the most recent checkpoint. **3. Deploy from checkpoint** `baseten train checkpoint deploy --job-id ` constructs a Truss `config.yaml` from the checkpoint, packages it as a deployment, and exposes an API endpoint. From there, the deployment behaves like any other model on Baseten. For more information, see the [Training Jobs overview](/training/overview). ## Request routing Each model gets a dedicated subdomain: `https://model-{model_id}.api.baseten.co/`. The URL path selects the deployment that handles a request. `/production/predict` targets the production environment, and `/development/predict` targets the development deployment. You can also target a deployment by ID or an environment by name. Baseten resolves the target from the URL and routes the request to an active replica. If the deployment has scaled to zero, Baseten starts a replica and holds the request until the model loads. The request uses the same endpoint whether the replica is warm or cold-started. Engine-based deployments expose an [OpenAI-compatible API](/reference/inference-api/chat-completions) at `/v1/chat/completions`. Use the OpenAI SDK with the deployment's base URL and a Baseten API key. Custom model deployments use the [predict API](/reference/inference-api/overview), which accepts and returns arbitrary JSON. For long-running workloads, [async requests](/inference/async) return a request ID immediately. An async request service queues the request. A background worker then calls your model and delivers the result through a webhook. Sync requests get priority when capacity is tight, so background work doesn't starve real-time traffic. ## Autoscaling Baseten's autoscaler matches replica count to concurrent request load, keeping each replica below its [concurrency target](/deployment/autoscaling/overview). When average load over the autoscaling window (60 seconds by default) crosses the target utilization (70% by default), the autoscaler adds replicas up to the configured maximum. When load drops, the autoscaler waits for `scale_down_delay` (900 seconds by default), then removes excess replicas at a pace capped by `max_scale_down_rate` (half of running replicas by default). The timer resets after each reduction, and the cycle repeats until the deployment reaches its target size. This staged reduction prevents scaling changes in response to brief traffic dips. Set [`min_replica`](/deployment/autoscaling/overview) to 0 for scale-to-zero: the deployment incurs no GPU cost when idle, but the next request triggers a cold start. Set `min_replica` to 1 or higher to keep warm capacity ready, trading cost for lower latency. ## Cold starts and the Baseten Delivery Network Loading model weights can dominate cold-start time, especially when weights reach hundreds of gigabytes. The [Baseten Delivery Network (BDN)](/development/model/bdn) caches model weights across storage, clusters, and nodes. On the first deployment, BDN mirrors model weights from the source repository to Baseten storage. Later cold starts do not depend on the original Hugging Face, S3, or GCS source. When a replica starts, the BDN agent fetches a weight manifest, downloads files through a cache shared by the cluster, and stores them in a node-level cache. BDN deduplicates identical files across models, so a fine-tune that shares files with its base model downloads only the difference. Later cold starts on the same node or cluster can reuse cached weights. Container image streaming also lets the model begin loading weights before the image download completes. BDN serves training jobs the same way. Mount weights and training data into your training container from any supported source, and BDN caches them so subsequent jobs start faster. ## Environments and promotion Use a development deployment with scale-to-zero and live reload for fast iteration. When the model is ready for stable traffic, promote the development deployment directly to a named [environment](/deployment/environments), such as production or staging. Promotion creates a published deployment from the current development state. Each environment has its own stable URL, autoscaling settings, and metrics. When you promote a new deployment, Baseten applies the environment's autoscaling settings and routes its endpoint to the new deployment according to its promotion settings. The endpoint URL stays constant, so your calling code doesn't need to change. After the promotion, Baseten handles the previous deployment according to the environment's cleanup strategy. Promoting a published deployment reuses its existing image. Re-promoting a previous published deployment for rollback also reuses that image. Promoting a development deployment creates a new published deployment from its current state and triggers an image build. To skip the development stage, push directly to an environment with `baseten model push --environment staging`. Only one promotion can be active per environment at a time, which prevents conflicting updates. See [Deployment concepts](/deployment/concepts) for the full set of resource and CI/CD options. These pieces work the same whether you deploy an existing model or train a new one, so the path from prototype to production stays consistent. ## Next steps > [card] [Deploy your first model](/development/model/build-your-first-model) > > Deploy a Hugging Face model with `config.yaml` and the Baseten CLI. > [card] [Training on Baseten](/training) > > Run a fine-tune or pre-train and deploy the checkpoint to an endpoint. # Autoscaling Source: https://docs.baseten.co/deployment/autoscaling/overview Configure autoscaling to dynamically adjust replicas based on traffic while minimizing idle compute costs. Autoscaling adjusts the number of **replicas** behind a deployment as request load changes. It adds replicas when traffic rises and removes them when traffic falls. Configure the minimum, maximum, and scaling thresholds to balance latency, capacity, and cost. Baseten [bills each running replica by the minute](/organization/billing). A deployment at zero replicas incurs no GPU charges, but starting and loading a new replica is billable. See [Startup times](/deployment/autoscaling/startup-times) for ways to reduce startup time. > [accordion] Reference > > Start with the default settings, then tune them for your model and traffic pattern. > > | Parameter | Default | Range | What it controls | > | - | - | - | - | > | Min replicas | 0 | ≥ 0 | Baseline capacity (0 = scale to zero). | > | Max replicas | 1 | ≥ 1 | Cost/capacity ceiling. | > | Autoscaling window | 60s | 10-3600s | Time window for traffic analysis. | > | Scale-down delay | 900s | 0-3600s | Wait time before removing idle replicas. | > | Max scale-down rate | 50% | 1-50% | Cap on replicas removed per scale-down step. | > | Concurrency target | 1 | ≥ 1 | Requests per replica before scaling. | > | Target utilization | 70% | 1-100% | Headroom before scaling triggers. | Use this page to choose autoscaling settings and understand how they interact. To apply settings to a live deployment from the Baseten dashboard or Management API, see [Scale a deployment](/deployment/manage/scaling). To change settings on a recurring or one-time schedule, see [Schedule autoscaling](/deployment/autoscaling/schedules). ## How autoscaling works The autoscaler averages concurrent requests over `autoscaling_window` (60 seconds by default). It divides that load by each replica's effective capacity (`concurrency_target` × `target_utilization_percentage`) and rounds up to calculate the desired replica count. Scale-up begins at the next decision. Scale-down waits for `scale_down_delay` and removes at most `max_scale_down_rate` of the running replicas in each step. The slower scale-down prevents brief traffic dips from repeatedly stopping and starting replicas. Use the simulator to see how the settings respond to different traffic patterns. Start with a scenario that demonstrates a cold start or scaling oscillation, or adjust the traffic and parameters yourself. The meters show idle capacity and queued requests. The simulator models requests waiting for capacity, not error responses from load shedding. See [Request lifecycle](/deployment/autoscaling/request-lifecycle#request-queuing-and-load-shedding). For example, set `concurrency_target` to 10 and `target_utilization_percentage` to 70%. Each replica then has an effective capacity of 7 concurrent requests (10 × 0.70). At an average of 25 concurrent requests, the autoscaler calculates ⌈25 / 7⌉ = 4 desired replicas and starts the additional capacity. In the simulator, further requests queue after the deployment reaches `max_replica`. Compare the **Desired** line on the [Replicas graph](/observability/metrics#replicas) against your current replica count to confirm your settings produce the scale you expect. When average load drops below the threshold, the autoscaler waits for `scale_down_delay` (900 seconds by default), removes replicas up to `max_scale_down_rate` (50% by default), and resets the timer. At the default rate, eight replicas reduce to four, then two, then one, with a full delay between steps. If traffic returns during the delay, the replicas stay active. Scale-down stops at `min_replica`. ## Replicas Each replica is an independent instance of the model on its own hardware. Replicas serve requests in parallel. You set the minimum and maximum replica counts, and the autoscaler adjusts capacity within those boundaries. **min_replica** `integer` (default: 0) The floor for your deployment's capacity. The autoscaler won't scale below this number. **Range:** ≥ 0 The default of 0 enables *scale-to-zero*: when no requests arrive for long enough, all replicas shut down and your deployment incurs no running-replica charges. When traffic returns, a synchronous request can wait while the deployment [cold-starts](/deployment/autoscaling/startup-times) a replica. During startup, [billing is per minute](/organization/billing) even though the replica isn't yet serving responses. > [!NOTE] > > Use `min_replica: 1` as a starting point when you need to avoid scale-from-zero latency. Use at least 2 when you also need replica redundancy. Replicas added during scale-up still cold-start. **max_replica** `integer` (default: 1) The ceiling for your deployment's capacity. The autoscaler won't scale above this number. **Range:** ≥ 1 This setting limits capacity and cost. When traffic exceeds the capacity of the maximum replicas, the autoscaler doesn't start more replicas. Synchronous requests can wait for an open slot, and load shedding can reject traffic if the queue grows too large. See [Request lifecycle](/deployment/autoscaling/request-lifecycle#request-queuing-and-load-shedding). The default of 1 caps the deployment at one replica. With the default `min_replica` of 0, the deployment can run zero or one replica, depending on demand. Estimate max replicas: $$ (peak\_requests\_per\_second / throughput\_per\_replica) + buffer $$ For high-volume workloads requiring guaranteed capacity, [contact Baseten](mailto:support@baseten.co) about reserved capacity options. ## Scaling triggers The autoscaler compares concurrent requests with a per-replica threshold. `concurrency_target` sets how many simultaneous requests a replica accepts. `target_utilization_percentage` reserves headroom by triggering scale-up before replicas reach that limit. Scale-up begins when: $$ load > replicas \times concurrency\_target \times target\_utilization $$ The following diagram shows a replica with `concurrency_target` of 8 and `target_utilization` of 50%, so the per-replica threshold sits at 4. The first four requests fill capacity within headroom; the fifth crosses the threshold, and the autoscaler provisions a second replica to absorb the overflow before the remaining slots saturate. **concurrency_target** `integer` (default: 1) How many requests each replica can handle simultaneously. This directly determines replica count for a given load. **Range:** ≥ 1 Given the current load, the autoscaler calculates desired replicas: $$ desired\_replicas = \lceil concurrent\_requests / (concurrency\_target \times target\_utilization) \rceil $$ *Concurrent requests* are requests sent to your model that haven't returned a response (for streaming, until the stream completes). [Async inference requests](/inference/async) are not included in this count. This count is exposed as [`baseten_concurrent_requests`](/observability/export-metrics/supported-metrics#baseten_concurrent_requests) in the metrics dashboard and metrics export. The right value depends on how your model uses hardware. Benchmark representative requests on your chosen instance type. A model that processes one request at a time may need a concurrency target of 1. Engines that batch requests may support higher targets, but setting the target beyond measured capacity adds per-replica queuing and latency. Higher concurrency uses fewer replicas but can increase per-replica queuing and latency. Lower concurrency uses more replicas but reduces queuing. When `predict` takes most of the request time, as in most GPU inference, keep the target at or below `predict_concurrency` so extra load spreads across replicas. When preprocessing or postprocessing waits on I/O, such as a file download, a target above `predict_concurrency` overlaps that work while `predict` stays serialized, and requests past that limit wait in the replica. See [`predict_concurrency`](/reference/truss-configuration#runtime) and [pre/post-processing](/development/model/model-class#pre/post-processing). **Starting points by model type:** Use these values to choose an initial benchmark. Your model, engine configuration, request shape, and hardware determine the final setting. | Model type | Starting concurrency | | - | - | | Standard Truss model | 1 | | vLLM / LLM inference | 32-128 | | SGLang | 32 | | Text embeddings (TEI) | 12 | | BEI embeddings | 12 | | Whisper (async batch) | 256 | | Image generation (SDXL) | 1 | For engine-specific guidance, see [Autoscaling engines](/engines/performance-concepts/autoscaling-engines). > [!NOTE] > > `predict_concurrency` (Truss `config.yaml`) limits only the requests that run `predict` at once. Preprocessing and postprocessing run outside that limit, so the right concurrency target depends on where your model spends its time. **target_utilization_percentage** `integer` (default: 70) Headroom before scaling triggers. The autoscaler scales when utilization reaches this percentage of the concurrency target, not when replicas are fully loaded. **Range:** 1-100% The effective threshold is: $$ concurrency\_target × target\_utilization $$ With a concurrency target of 10 and utilization of 70%, scaling triggers at 7 concurrent requests (10 × 0.70), leaving 30% headroom for absorbing spikes while new replicas start. As a starting point, lower values such as 50% to 60% provide more headroom for spikes but cost more. Higher values such as 80% or more use capacity more fully, but leave less headroom while new replicas start. > [!WARNING] > > Target utilization is **not** GPU utilization. It measures request slot usage relative to your concurrency target, not hardware utilization. ## Scaling dynamics Three settings control scaling speed. `autoscaling_window` determines how much traffic history each decision uses. `scale_down_delay` keeps replicas active for a period after load drops. `max_scale_down_rate` limits how many replicas each scale-down step removes. The diagram shows traffic falling to zero and a replica stopping after the delay expires. **autoscaling_window** `integer` (default: 60) How far back (in seconds) the autoscaler looks when measuring traffic. Traffic is averaged over this window to make scaling decisions. **Range:** 10-3600 seconds A 60-second window smooths out momentary spikes by averaging load over the past minute. Start with 30 to 60 seconds for bursty traffic. Start with 2 to 5 minutes when you want to filter short-lived fluctuations. Adjust the window against your observed traffic and replica startup time. **scale_down_delay** `integer` (default: 900) How long (in seconds) the autoscaler waits after load drops before removing replicas. **Range:** 0-3600 seconds When load drops, the autoscaler starts a countdown. If load stays low for the full delay, it removes replicas in steps, each capped by `max_scale_down_rate`, with a fresh delay between steps. If traffic returns before the countdown finishes, the replicas stay active and the countdown resets. If replicas repeatedly scale up and down, increase this value first. [BIS-LLM](/engines/bis-llm/overview) deployments default to 300 seconds. **max_scale_down_rate** `integer` (default: 50) The maximum percentage of running replicas the autoscaler can remove in one scale-down step. **Range:** 1-50% Each time a `scale_down_delay` elapses, the autoscaler removes at most this percentage of running replicas. The default of 50% produces the halve-and-wait pattern described above. Lower values release capacity more gradually, which keeps more replicas warm when traffic tends to rebound shortly after it drops. BIS-LLM deployments default to 20% because replicas hold KV cache that's expensive to rebuild. > [!TIP] > > A **short window** with a **long delay** gives you fast scale-up while maintaining capacity during temporary dips. This is a good starting configuration for most workloads. ## Development deployments Development deployments are designed for iteration, not production traffic. Replicas are fixed at 0-1 to match the [`baseten model push --watch`](/reference/cli/baseten/model#push) workflow, where you're testing changes on a single instance rather than handling concurrent users. You can still adjust timing and concurrency settings. | Setting | Value | Modifiable | | - | - | - | | Min replicas | 0 | No | | Max replicas | 1 | No | | Autoscaling window | 60 seconds | Yes | | Scale-down delay | 900 seconds | Yes | | Concurrency target | 1 | Yes | | Target utilization | 70% | Yes | To configure the full set of autoscaling settings, [promote the deployment to an environment](/deployment/manage/lifecycle#promote-to-an-environment). ## Troubleshooting See [Autoscaling troubleshooting](/troubleshooting/deployments#autoscaling-issues) for help with oscillation, slow scale-up, and unexpected costs. ## Next steps > [card] [Traffic patterns](/deployment/autoscaling/traffic-patterns) > > Identify your traffic pattern and get recommended starting settings. > [card] [Startup times](/deployment/autoscaling/startup-times) > > Understand cold starts and how to minimize their impact. > [card] [API reference](/reference/management-api/deployments/autoscaling/updates-a-deployments-autoscaling-settings) > > Complete autoscaling API documentation. > [card] [Engine-specific autoscaling](/engines/performance-concepts/autoscaling-engines) > > Recommended settings for BEI and Engine-Builder-LLM with dynamic batching. # Request lifecycle Source: https://docs.baseten.co/deployment/autoscaling/request-lifecycle How Dedicated Inference routes and queues requests, applies timeouts, and returns errors. When you call an OpenAI-compatible endpoint or the [predict API](/inference/calling-your-model), Baseten authenticates and routes the request before selecting a replica. For a Truss model with custom code, the replica runs your `predict` function after these steps. The request path explains where latency occurs, what each status code means, and how to debug a failed request. ## How a request reaches your model Your request first hits Baseten's inference gateway, which authenticates it against your [API key](/organization/api-keys). If authentication fails, the gateway rejects the request before it reaches any model infrastructure: `401` when the request has no credentials, `403` when the key is invalid. A standard endpoint also returns `403` when the environment [requires a regional endpoint](/deployment/regional-environments#compare-regional-and-standard-endpoints). Once authenticated, the request moves to the routing layer. Baseten selects a workload plane where the deployment runs, then distributes traffic among the deployment's replicas in that plane. The [concurrency target](/deployment/autoscaling/overview#param-concurrency-target) defines how many simultaneous requests each replica should handle and informs autoscaling decisions. When routing succeeds, Baseten forwards the request to the deployment, and a replica runs inference. For deployments that use the predict API, your `predict` function executes here. The response flows back through the same path to the client. For most requests, the routing overhead is negligible compared to your model's inference time. The sections below cover what happens when this straightforward path breaks down: when no replica is available, when replicas reach capacity, and when requests fail partway through. ## What happens when no replica is available If your deployment has scaled to zero, no replica is ready to receive a request. By default, Baseten parks a synchronous request at the routing layer and waits for a replica to become available. Once a replica is ready, Baseten forwards the parked request, and the model processes it normally. From the client's perspective, this wait adds to the normal inference time. This parking behavior makes [scale-to-zero](/deployment/manage/scaling#scale-to-zero) practical for deployments that queue requests. You don't need to retry a request because your deployment was idle. The request waits for a replica, but only until the parking timeout expires. The parking timeout uses the same configured duration as the predict timeout, which is 1200 seconds by default. If no replica becomes available by then, the request fails with a `500`. To avoid scale-from-zero latency, keep [minimum replicas](/deployment/autoscaling/overview#param-min-replica) greater than zero. [Async requests](/inference/async) follow a different path. The `/async_predict` endpoint accepts the request into the async service without waiting for a replica. The service dispatches the request later and applies its own [inference retry policy](/inference/async#inference-retries). For strategies to reduce cold start latency, including warm replicas, pre-warming, and the Baseten Delivery Network, see [Startup times](/deployment/autoscaling/startup-times). ## Request queuing and load shedding Even when replicas are running, they can fill up. By default, incoming synchronous requests wait for an open replica slot. The autoscaler can add replicas within your configured limits. Load shedding prevents the queue from growing without bound. Baseten rejects a new request with a `429` when queued payloads create memory pressure or the queue crosses its soft limit. Baseten returns a `529` when the queue crosses its hard limit. Retry either response with exponential backoff and jitter. Your client can't distinguish time spent in the queue from time spent running inference. Set a client-side timeout that matches your latency requirements. If queuing persists, increase [max replicas](/deployment/autoscaling/overview#param-max-replica) when the deployment reaches its replica ceiling. Raise the [concurrency target](/deployment/autoscaling/overview#param-concurrency-target) only when benchmarks show that each replica can handle more simultaneous requests. ### Request backpressure policies A request backpressure policy controls what happens to a new synchronous request when no replica has an open request slot: | Policy | Behavior | | - | - | | **Queue on full** | Waits for an open replica slot. This is the default. | | **Reject on full** | Returns `529` immediately with the error code `request_backpressure_rejected`. | Use **Queue on full** when your clients can tolerate variable latency and you want the deployment to absorb short traffic spikes. Use **Reject on full** when clients have strict latency limits or can retry requests elsewhere. Rejected requests don't reach a replica, so they don't contribute to concurrency-based autoscaling. Set the [target utilization percentage](/deployment/autoscaling/overview#param-target-utilization) below 100% so admitted traffic can trigger scaling before every request slot fills. If the deployment has scaled to zero, the first request triggers a replica start in the background. That request and later requests receive `529` until a replica is ready. Retry the request with exponential backoff and jitter. The reject policy applies only to synchronous HTTP requests; `/async_predict` keeps its own queue and retry behavior. Request backpressure policies aren't available for WebSocket, gRPC, Chain, or [Baseten Inference Stack](/engines/bis-llm/overview) deployments. Baseten Inference Stack uses separate router-level capacity controls that return `429` based on request shape, such as the number of cached tokens and tokens that aren't cached. Policy rejections appear as `5xx` responses in the [Inference volume](/observability/metrics#inference-volume) chart. They also appear in [deployment logs](/observability/logs#scope-by-environment-or-deployment) with the error code `request_backpressure_rejected`. ### Backpressure configuration > [!NOTE] > > Request backpressure policies are rolling out gradually. If the **Request backpressure policy** setting isn't visible in your deployment or environment settings, [contact support](mailto:support@baseten.co) to enable it for your organization. Choose a standalone deployment to configure only that deployment. Choose an environment to keep the policy with the environment as you promote new deployments into it. You can't change the policy while a rolling deployment is in progress. Wait for the promotion to finish, or cancel the promotion before changing the policy. > [tab] Baseten CLI > > **To configure a request backpressure policy**: > > ```bash Command theme={"system"} > baseten model deployment update-request-backpressure \ > --model-id \ > --deployment-id \ > --policy reject-on-full > ``` > > ```txt Output theme={"system"} > Request backpressure policy: reject-on-full > ``` > > To restore queuing, pass `--policy queue-on-full`. Pass `--policy null` to clear an explicit policy and use the default. > > For more information, see > [`deployment update-request-backpressure`](/reference/cli/baseten/model#update-request-backpressure). > [tab] UI > > **To configure a request backpressure policy**: > > 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar. > 2. Select your model. > 3. Under **Deployments** or **Environments**, select the deployment or environment you want to configure. > 4. Under **Request handling**, choose **Manage**. > 5. For **Request backpressure policy**, select **Queue on full (default)** or **Reject on full**. > 6. Choose **Update**. > [tab] Management API > > **To update a standalone deployment's request backpressure policy**: > > 1. Get the model ID and deployment ID you want to configure. > > 2. Update the deployment's request backpressure settings: > > ```bash Request theme={"system"} > curl -X PATCH "https://api.baseten.co/v1/models/abc123/deployments/def456/request_backpressure_settings" \ > -H "Authorization: Bearer $BASETEN_API_KEY" \ > -H "Content-Type: application/json" \ > -d '{"policy": "REJECT_ON_FULL"}' > ``` > > ```json Response theme={"system"} > { > "policy": "REJECT_ON_FULL" > } > ``` > > 3. Confirm that the response contains `"policy": "REJECT_ON_FULL"`. > > **To update an environment's request backpressure policy**: > > This procedure requires the external `jq` command to display only the request backpressure settings from the response. > > 1. Get the model ID and environment name you want to configure. > > 2. Update the environment settings: > > ```bash Request theme={"system"} > curl -X PATCH "https://api.baseten.co/v1/models/abc123/environments/production" \ > -H "Authorization: Bearer $BASETEN_API_KEY" \ > -H "Content-Type: application/json" \ > -d '{"request_backpressure_settings": {"policy": "REJECT_ON_FULL"}}' \ > | jq '.environment.request_backpressure_settings' > ``` > > ```json Response theme={"system"} > { > "policy": "REJECT_ON_FULL" > } > ``` > > 3. Confirm that the response contains `"policy": "REJECT_ON_FULL"`. > > **To set a request backpressure policy when you create an environment**: > > This procedure requires the external `jq` command to display only the request backpressure settings from the response. > > 1. Get the model ID for the new environment. > > 2. Include `request_backpressure_settings` in the create request: > > ```bash Request theme={"system"} > curl -X POST "https://api.baseten.co/v1/models/abc123/environments" \ > -H "Authorization: Bearer $BASETEN_API_KEY" \ > -H "Content-Type: application/json" \ > -d '{"name": "staging", "request_backpressure_settings": {"policy": "REJECT_ON_FULL"}}' \ > | jq '.request_backpressure_settings' > ``` > > ```json Response theme={"system"} > { > "policy": "REJECT_ON_FULL" > } > ``` > > 3. Confirm that the response contains `"policy": "REJECT_ON_FULL"`. > > To restore queuing for a standalone deployment, send `{"policy": "QUEUE_ON_FULL"}`. For an environment, send `{"request_backpressure_settings": {"policy": "QUEUE_ON_FULL"}}`. In either payload, replace `"QUEUE_ON_FULL"` with `null` to clear an explicit policy and use the default effective policy. ## Internal retries When a replica returns a retryable status such as `502`, `503`, or `504`, the routing layer retries the request with exponential backoff. Connection failures use the same retry path. Retries stop when they reach the configured maximum number of prediction attempts or the request deadline, whichever comes first. The routing layer doesn't apply these retries to async requests. The async service has a separate [inference retry policy](/inference/async#inference-retries). From your client's perspective, retries add latency instead of immediately returning an error. Check the `X-BASETEN-MODEL-PREDICTION-ATTEMPTS` response header when a request takes longer than expected. A value greater than 1 confirms that Baseten retried the request. Baseten can temporarily suppress retries when the routing layer is under pressure. If a sticky-session request returns a `503`, the retry routes to a different replica. ## Timeouts The **predict timeout** controls how long a sync request can take from the moment it's forwarded to a replica until a response must be returned. If your model's inference exceeds this window, the request is cancelled and the client receives a `504`. The server-side default is 1200 seconds (20 minutes). If you need requests to fail faster than that, set a client-side timeout in your HTTP client. The **async predict timeout** is 3600 seconds (1 hour) per inference attempt. A timeout produces a retryable `504` within the async service. Baseten follows the request's [async inference retry policy](/inference/async#inference-retries). After the service exhausts those attempts, it marks the request as failed with a `MODEL_PREDICT_TIMEOUT` error status and sends the error payload to your webhook. For a parked synchronous request, the **parking timeout** controls how long it waits when the deployment has no ready replicas. It uses the same configured duration as the synchronous predict timeout, but the two timeouts apply separately. If routing succeeds before the parking timeout expires, the full predict timeout starts when Baseten forwards the request to a replica. With the default settings, a synchronous request can spend up to 1200 seconds parked and then up to another 1200 seconds running inference. For **streaming responses**, timeouts behave differently because the HTTP headers, including the `200` status code, are sent when the stream begins. If the timeout expires mid-stream, the stream stops and the connection closes without an error code, since the status was already written. Most HTTP clients surface this as a connection reset or incomplete response rather than a timeout error. ## Requests during a promotion You can [promote a deployment](/deployment/manage/lifecycle#promote-to-an-environment) while the current one is serving live traffic without dropping concurrent requests. As traffic shifts to the new deployment, each replica of the previous one stops accepting new requests and finishes the work it's already doing before shutting down. There's nothing to configure: draining happens automatically, whether the promotion replaces the deployment immediately or through a [rolling deployment](/deployment/rolling-deployments). Draining replicas stay alive for up to one hour. This window covers a single in-flight attempt: synchronous prediction has a 1200-second default timeout, and an async prediction attempt has a 3600-second timeout. The async service schedules any retry as a separate attempt. If your model runs its own shutdown logic, such as flushing state in a custom server, it receives a `SIGTERM` when its replica starts draining. ## HTTP status codes This table summarizes the status codes produced by the routing and lifecycle behavior on this page. Request-side errors such as `400` and `413` come from other layers. > [!NOTE] > > For what each error means, how to tell a model failure from a Baseten-side issue, and where to look next, see [Inference errors](/inference/errors). | Code | Meaning | When it occurs | What to do | | - | - | - | - | | `200` | Success | Normal predict response. | None. | | `201` | Created | Async predict request queued successfully. | Use a [webhook](/inference/async#webhooks) to receive the output. Poll the status endpoint only to monitor the request lifecycle. | | `401` / `403` | Unauthorized / Forbidden | Missing credentials (`401`), an invalid API key (`403`), or use of the standard endpoint for an environment that requires a regional endpoint (`403`). | Check your [API key](/organization/api-keys). For a regional environment, use its [regional endpoint](/deployment/regional-environments#regional-endpoint-formats). | | `429` | Too Many Requests | No replica slot was available (`CAPACITY_EXCEEDED`), or load shedding detected memory pressure or the soft queue limit. | Retry with exponential backoff. If persistent, increase [max replicas](/deployment/autoscaling/overview#param-max-replica) or tune the [concurrency target](/deployment/autoscaling/overview#param-concurrency-target). | | `499` | Client Closed Request | Client disconnected before the response was written. | No server-side action needed. Review client-side timeout configuration if unexpected. | | `500` | Internal Server Error | A sync request's parking timeout expired before a replica became available. | Retry with exponential backoff. If persistent, keep [minimum replicas](/deployment/autoscaling/overview#param-min-replica) greater than zero or reduce [startup time](/deployment/autoscaling/startup-times). | | `502` | Bad Gateway | The model became unavailable during inference. | Retry. If persistent, check model logs for crashes or errors in your `predict` function. | | `503` | Service Unavailable | The routing layer couldn't find a replica endpoint, typically during a deployment rollout or immediately after a replica failure. | Retry. If persistent, check deployment status in the Baseten dashboard. | | `504` | Gateway Timeout | The request exceeded the server-side predict timeout (1200 seconds). | Optimize your model's inference speed or use [async inference](/inference/async) for longer work. | | `529` | Overloaded | The **Reject on full** policy rejected the request because no replica slot was available, or load shedding detected that the queue crossed its hard limit. Policy rejections include the error code `request_backpressure_rejected`. | Retry with exponential backoff and jitter. If persistent, increase [max replicas](/deployment/autoscaling/overview#param-max-replica), tune the [concurrency target](/deployment/autoscaling/overview#param-concurrency-target), or use **Queue on full** when clients can tolerate the added latency. | > [!NOTE] > > A `500` from a sync request during a cold start can mean the parking timeout expired before a replica finished starting. Retry with exponential backoff. If this recurs, keep [minimum replicas](/deployment/autoscaling/overview#param-min-replica) greater than zero or reduce [startup time](/deployment/autoscaling/startup-times). ## Request cancellation When a client disconnects before the response is written, the routing layer detects the closed connection and cancels the in-flight work. The server logs this as a `499`. In the common case, such as a user closing a browser tab or a client-side timeout firing, this is harmless and the `499` is informational rather than an error. If a client disconnects during a long generation and the model keeps running, you're paying for GPU time that produces tokens nobody reads. Baseten cancels in-flight work automatically so this doesn't happen. When the routing layer detects a disconnect, it signals the inference engine, which aborts the running request and frees GPU resources. This works across engines including TRT-LLM and vLLM. If you're using a custom model server, you can implement cancellation yourself using Truss request objects. See [Request handling](/development/model/streaming-and-endpoints#request-handling) for code examples. ## Next steps > [card] [Startup times](/deployment/autoscaling/startup-times) > > Reduce cold start latency with warm replicas and pre-warming strategies. > [card] [Autoscaling](/deployment/autoscaling/overview) > > Configure concurrency targets, replica counts, and scaling dynamics. > [card] [Async inference](/inference/async) > > Fire-and-forget inference with webhook delivery. > [card] [Troubleshooting](/troubleshooting/deployments) > > Diagnose common deployment issues including autoscaling problems. # Schedule autoscaling for traffic Source: https://docs.baseten.co/deployment/autoscaling/schedules Change autoscaling settings on a recurring or one-time window instead of running an external cron job. Autoscaling responds to traffic that has already arrived, so requests at the start of an increase can wait behind a [cold start](/deployment/autoscaling/startup-times) while new replicas start. When your traffic follows known clock times, you can change the autoscaling settings before the traffic arrives rather than after. An *autoscaling schedule* is a window on a model [environment](/deployment/environments), recurring or one-time, carrying its own autoscaling settings. While the window is active, the environment runs the schedule's settings. Outside every window, it runs the environment's default configuration. Schedules do this natively, so you don't need an external [cron job](/deployment/autoscaling/traffic-patterns#scheduled-pre-warming) or [GitHub Action](/deployment/ci-cd) calling the [Management API](/deployment/manage/scaling) on a timer. ## How schedules work Autoscaling defaults to the configuration values you set on the environment. To vary those values by time of day or day of week, define up to 10 schedules, each pairing a recurring window with its own set of autoscaling settings. The schedule whose window covers the current time is the one in effect. A schedule's window runs on one of three cadences: * **Daily**: one window a day, between a start and end time, on the days you choose, such as `09:00` to `18:00` on Monday through Friday. * **Hourly**: a window in every hour, optionally limited to a range of hours, such as `:55` to `:20` during hours `09` to `17`. * **One time**: a single window between two absolute date-times, such as September 1 at `09:00` to `13:00`. Use one-time schedules for planned events like launches, load tests, and migrations. The window does not repeat, and the schedule stops applying when it ends. Windows include their start and exclude their end: in a `09:00` to `17:00` window, `09:00` is inside and `17:00` is not. A recurring end time at or before the start wraps past midnight, so a `22:00` to `06:00` window covers the overnight hours. Every schedule on an environment shares one timezone, set once for the whole set. Recurring times are local to it; one-time windows accept any timezone offset and convert to UTC. ## Configure a schedule You configure schedules per environment, from the same dialog as the rest of your autoscaling settings. **To add an autoscaling schedule**: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar. 2. Select your model, then select the deployment for the environment you want to schedule. 3. Choose **Configure autoscaling**. 4. For **Config to edit**, choose **Schedules**. 5. Choose **Add schedule**, then set its window and replicas: * For **Frequency**, choose **Daily**, **Hourly**, or **One time**. * For a daily or hourly schedule, select the days it runs under **On days**, then set **Starts at** and **Ends at**. For an hourly schedule, use **Limit to hours** to restrict which hours it runs in. * For a one-time schedule, set the **Schedule window** start and end date-times. * Set **Min** and **Max** replicas for the window. To override the remaining autoscaling settings, choose **More options**. 6. Choose **Save**. If the new schedule's window covers the current time, the environment switches to it as soon as you save. An **Active** badge marks whichever configuration is running. > [!NOTE] > > A start time is the earliest the settings change, not a promise that replicas are ready to serve. If the deployment can't take the change right then, because it's deploying, loading a model, mid [rolling promotion](/deployment/rolling-deployments), or migrating, Baseten applies it as soon as it can. ## Validation Saving fails if any schedule breaks one of these rules: * **Count**: up to 10 schedules per environment. Expired one-time schedules don't count. * **Window length**: at least 5 minutes, long enough to scale up and back down. A one-time window can span at most 48 hours, including back-to-back chains of one-time schedules. * **Overlap**: two windows can't cover the same time. * **Gaps**: windows either touch exactly or leave at least 5 minutes between them. * **Coverage**: schedules can't fill the entire week, so the default configuration always has time to run. * **Timezone**: one shared timezone per environment. * **One-time start**: a new one-time schedule must start in the future. You can still edit an existing one-time schedule after its window starts. * **One-time horizon**: a one-time window can end at most 90 days in the future. Windows that touch exactly hand off directly. A `05:40` to `09:00` window can hand straight to a `09:00` to `18:00` window with no gap and no return to the default in between. ## Edit or remove a schedule Changes to the schedule that's currently running apply immediately, not at its next start. Disabling or removing it falls back to the default, or to another schedule whose window covers the current time. **To edit, rename, duplicate, disable, or remove a schedule**: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar. 2. Select your model, then select the deployment for the environment. 3. Choose **Configure autoscaling**. 4. For **Config to edit**, choose **Schedules**. 5. Expand the schedule and make your changes: * To rename, duplicate, or remove it, use the actions menu on its header. * To pause it without removing it, use the **Enabled** switch. 6. Choose **Save**. A disabled schedule still has to pass validation, so you can re-enable it later. A one-time schedule stays on the environment after its window passes; remove it when you no longer need it. ## Manage schedules from the Baseten CLI The [`baseten model environment autoscaling-schedule`](/reference/cli/baseten/model) commands manage the same schedule collection as the dashboard and API. ### Create a schedule ```bash Command theme={"system"} baseten model environment autoscaling-schedule create \ --model-id \ --environment production \ --name "Weekday business hours" \ --cadence daily \ --weekdays monday,tuesday,wednesday,thursday,friday \ --start-hour 9 \ --end-hour 18 \ --min-replica 2 \ --max-replica 8 \ --timezone America/Los_Angeles ``` ```txt Output theme={"system"} Created autoscaling schedule Weekday business hours ``` For a one-time schedule, use `--cadence one-time` with `--start-at` and `--end-at` instead of the weekday and hour flags. The `--cadence` flag accepts `daily`, `hourly`, or `one-time`. For the full flag set, see [`autoscaling-schedule create`](/reference/cli/baseten/model#create). ### Update a schedule ```bash Command theme={"system"} baseten model environment autoscaling-schedule update \ --model-id \ --environment production \ --schedule-name "Weekday business hours" \ --max-replica 12 ``` Target a schedule by `--schedule-id` or `--schedule-name`. Changing `--cadence` requires the new shape's timing flags, since the old timing doesn't carry over. For the full flag set, see [`autoscaling-schedule update`](/reference/cli/baseten/model#update). ### Delete a schedule ```bash Command theme={"system"} baseten model environment autoscaling-schedule delete \ --model-id \ --environment production \ --schedule-name "Weekday business hours" ``` ```txt Output theme={"system"} Deleted autoscaling schedule VqmMZqO ``` For the full flag set, see [`autoscaling-schedule delete`](/reference/cli/baseten/model#delete-2). ## Manage schedules from the API The [Update environment](/reference/management-api/environments/update-an-environments-settings) endpoint manages the same schedule collection. Each entry in `schedules` is a complete create or replacement: include a schedule's `id` to replace it, or omit `id` to create a new one. Schedules you leave out of the list stay unchanged, and deletion stays explicit through `delete_schedules`. ```json PATCH /v1/models/{model_id}/environments/{env_name} theme={"system"} { "autoscaling_schedule_settings": { "timezone": "America/Los_Angeles", "schedules": [ { "cadence": "ONE_TIME", "name": "Launch day scale-up", "enabled": true, "start_at": "2026-09-01T16:00:00Z", "end_at": "2026-09-01T20:00:00Z", "autoscaling_settings": { "min_replica": 4, "max_replica": 16, "autoscaling_window": null, "scale_down_delay": null, "concurrency_target": null, "target_utilization_percentage": null, "target_in_flight_tokens": null, "max_scale_down_rate": null } } ] } } ``` The `cadence` field selects the timing shape: `DAILY` and `HOURLY` take the weekday and time-of-day fields, while `ONE_TIME` takes `start_at` and `end_at`. Every `autoscaling_settings` field is required; set a field to `null` to follow the environment's current value. One-time date-times must include a timezone offset, and Baseten converts them to UTC. The PATCH response reports the requested state, and schedule application reconciles asynchronously. Poll [Get environment](/reference/management-api/environments/get-an-environments-details) to confirm the applied settings. ## Next steps > [card] [Traffic patterns](/deployment/autoscaling/traffic-patterns) > > Match autoscaling settings to the shape of your traffic. > [card] [Startup times](/deployment/autoscaling/startup-times) > > Shrink the time between a scale-up and a served request. > [card] [Environments](/deployment/environments) > > Understand the environments that schedules attach to. > [card] [Scale a deployment](/deployment/manage/scaling) > > Apply autoscaling changes to a live deployment from a script. # Startup times Source: https://docs.baseten.co/deployment/autoscaling/startup-times The phases of replica startup, how to measure startup time, and the model and config changes that reduce it. *Startup time* is the time a new replica takes from creation until it passes its readiness check and can accept traffic. Every new replica goes through startup, whether it's the first replica of a deployment scaled to zero or one the autoscaler adds under load. A synchronous request that arrives while a deployment has no ready replicas waits for startup to finish, so startup time becomes part of that request's latency. ## Startup phases A new replica runs these phases in order. The **Startup time** card uses the same phase names. Total startup time runs from replica creation to ready, so it also includes short gaps between phases, such as container sandbox setup. | Phase | Starts | Ends | What drives its duration | | - | - | - | - | | Acquiring resources | Baseten creates the replica. | The replica is scheduled onto a node. | Availability of the requested instance type. | | Pulling image | The node starts pulling your container image. | The image is available on the node. | Image size. A replica on a node that already has your image skips this phase. | | Pulling weights | The node starts mounting the [Baseten Delivery Network (BDN)](/development/model/bdn) weights volume. | All weight files are on the node and mounted. | Weight size and which BDN cache tier serves them. | | Loading model | The node creates your model container. | The replica passes its readiness check. | Your setup code. The container starts, the model server starts, and your `load()` method runs. For inference engines like vLLM and SGLang, this includes loading weights into GPU memory, capturing CUDA graphs, compiling kernels with `torch.compile`, and profiling the KV cache. | **Pulling weights** appears only when BDN delivers the weights. Weights that your code downloads in `load()`, or through the legacy `model_cache` key, count toward **Loading model** instead. For Truss-built images, Baseten can stream the container image: the node starts your container once the files it needs at startup arrive, and the rest of the image downloads during **Loading model**. Deployment logs show "streaming-enabled image" when streaming is active. ## Measure startup time The **Startup time** card on the environment and deployment pages shows how long your replicas take to start. The environment page aggregates startups across every deployment that served the environment during the selected range. The deployment page shows startups for that deployment only. Choose a range of the last 24 hours or 7 days. The last 30 days range appears once 30 days of startup history are available. The card shows: * **Startup duration:** the mean total duration, from replica creation to ready, with a bar for each of p50, p90, p95, and p99, broken down by phase. The percentiles describe your startups better than the mean. Select a phase in the legend to highlight it in the chart and filter the history table. * **Startup history:** one row per replica startup, with its outcome, a timeline of its phases, and its start and ready times. Filter by outcome or phase, search by replica or deployment, and sort by outcome, duration, or start time. Each startup has one of these outcomes: * **Succeeded:** the replica became ready. * **Failed:** the replica stopped before it became ready, for example from an exception in `load()`, running out of memory, or an image pull error. * **Stalled:** the replica didn't become ready within 30 minutes. For a failed or stalled startup, the timeline shows the last phase the replica reached. Start there, then open the [logs](/observability/logs) for that time window for detail. A failure in **Pulling image** points to the image or registry. A failure in **Loading model** points to your `load()` code or to memory: an out-of-memory error during load needs a larger instance or smaller weights. The card counts only a replica's first startup. A container that crashes and restarts appears in the [Restarts](/observability/metrics#restarts) chart on the Metrics tab instead. ## Diagnose slow startups Find the phase that dominates your startups, then apply the matching fix: | Signal | Likely cause | What to do | | - | - | - | | Long **Acquiring resources** | Waiting for capacity | Choose a more available instance type. If this phase stays long, [contact support](mailto:support@baseten.co). | | Long or frequent **Pulling image** | Large image, or replicas landing on nodes without your image | Build a [smaller image](#smaller-container-images). | | Long **Pulling weights** | Large weights or cache misses | [Filter and pin your weights](#faster-weight-loading), or quantize the model. | | Long **Loading model** | Heavy setup code, or weights downloaded in `load()` | [Speed up model loading](#faster-model-loading). | | Many rows in **Startup history** | Autoscaling churn or scale-to-zero cycling | Tune your [autoscaling settings](/deployment/autoscaling/overview) so replicas start less often. | | Wide gap between p50 and p99 | Some startups hit cold caches | Your p50 startups reuse cached images and weights while your p99 startups don't. Enable BDN and reduce your image size before you optimize `load()`. | When p50 and p99 are both high and **Loading model** dominates, image and weight caching can't help: the time goes to work that runs on every startup. Optimize `load()`, and if your model uses `torch.compile`, enable [compilation caching](#compilation-caching). ## Reduce startup time Startup time drops most when you reduce the duration of the phase that dominates it. ### Smaller container images A smaller image pulls faster on a node that doesn't have it cached: * Remove unused entries from `requirements` and `system_packages` in your config. * Deliver weights through BDN instead of building them into the image. ### Faster weight loading BDN runs automatically on engine-builder deployments. On any other deployment, turn it on by adding a [`weights`](/development/model/bdn) block to your config: ```yaml config.yaml theme={"system"} weights: - source: "hf://Qwen/Qwen3-8B@" mount_location: "/models/qwen" allow_patterns: ["*.safetensors", "*.json"] ``` * Use `allow_patterns` and `ignore_patterns` to skip files your model doesn't serve, such as `.bin` checkpoints when `.safetensors` files exist. * Pin a revision (`@`) so cached weights stay valid across deployments. * If you use `model_cache`, run `truss migrate` to move to BDN. See [Cached weights](/development/model/model-cache). Quantized weights (FP8 or FP4) reduce both download and load time. ### Faster model loading **Loading model** covers everything in `load()`, so you control most of it: * Don't download files in `load()`. Move them to a `weights` block so BDN caches them and the time moves to **Pulling weights**. * Load weights from `safetensors` files instead of pickled checkpoints. * Move work that doesn't need to happen before the first request out of `load()`, or into build time. * Keep [custom health checks](/development/model/health-checks#custom-health-check-logic) from blocking readiness on long warm-up work. ### Compilation caching `torch.compile` creates artifacts while a replica starts. [Torch compile caching](/development/model/runtime-caching#torch-compile-caching), built on [b10cache](/development/model/runtime-caching), persists those artifacts so a new replica can reuse them instead of compiling from scratch. Benchmark your deployment to measure the effect on startup time. ### Keep startup off the request path To keep requests from waiting on startup, set [`min_replica`](/deployment/autoscaling/overview#param-min-replica) to 1 or higher so a replica is always running, or raise it ahead of known traffic spikes, using your p99 startup time as the lead time. See [Traffic patterns](/deployment/autoscaling/traffic-patterns#pre-warming-for-predictable-bursts) for pre-warming. ## Next steps * [Metrics](/observability/metrics): Response time, replica, and restart charts that complement the **Startup time** card. * [Request lifecycle](/deployment/autoscaling/request-lifecycle): What happens to requests during replica startup, including queuing and timeout behavior. * [Autoscaling](/deployment/autoscaling/overview): Control how often replicas start and how many stay running. * [Billing and usage](/organization/billing): How startup time is metered. * [Troubleshooting](/troubleshooting/deployments#autoscaling-issues): Diagnose slow or failed startups. # Traffic patterns Source: https://docs.baseten.co/deployment/autoscaling/traffic-patterns Identify your traffic pattern and configure autoscaling settings to match. Different traffic patterns require different autoscaling configurations. Identify your pattern below for recommended starting settings. > [!NOTE] > > These are **starting points**, not final answers. Monitor your > deployment's performance and adjust based on observed behavior. See > [Autoscaling](/deployment/autoscaling/overview) for parameter details. ## Identify your pattern Not sure which pattern you have? Check your metrics: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar, then select your model. 2. Choose the **Metrics** tab. 3. Look at **Inference volume** and **Replicas** over the past week. 4. Compare to the patterns below. | You see... | Your pattern is... | | - | - | | Frequent small spikes that quickly return to baseline | [Jittery](#jittery-traffic) | | Sharp jumps that stay high for a while | [Bursty](#bursty-traffic) | | Long flat periods with occasional large bursts | [Scheduled](#scheduled-traffic) | | Gradual rises and falls, smooth curves | [Steady](#steady-traffic) | > [!NOTE] > > If traffic combines a daily pattern with sharp bursts, tune for the bursts, then measure cost during steady periods. ## Jittery traffic Small, frequent spikes that quickly return to baseline. ### Characteristics * Traffic repeatedly rises to at least twice its baseline, then quickly returns. * Spikes are short-lived and return to baseline quickly. * Often not real load growth, just temporary surges causing overreaction. ### Common causes * Consumer products with intermittent usage bursts. * Traffic splitting or A/B testing with low percentages. * Polling clients with synchronized intervals. ### Recommended settings | Parameter | Value | Why | | - | - | - | | Autoscaling window | **2-5 minutes** | Smooth out noise, avoid reacting to every spike | | Scale-down delay | **300-600s** | Moderate stability | | Target utilization | **70%** | Default is fine | | Concurrency target | Benchmarked value | Start conservative | A longer autoscaling window averages out the jitter so the autoscaler doesn't react to every small spike. This trades reaction speed for stability when the spikes aren't sustained load increases. > [!TIP] > > If you're still seeing oscillation with these settings, increase the scale-down delay before lowering target utilization. ## Bursty traffic ### Characteristics * Traffic **jumps sharply** (2x+ within 60 seconds). * Stays high for a sustained period before dropping. * Queuing and latency increase while replicas start. ### Common causes * Daily morning ramp-up (users starting their day). * Marketing events, product launches, viral moments. * Top-of-hour scheduled jobs or cron-triggered traffic. ### Recommended settings | Parameter | Value | Why | | - | - | - | | Autoscaling window | **30-60s** | React quickly to genuine load increases | | Scale-down delay | **900s+** | Handle back-to-back waves without thrashing | | Target utilization | **50-60%** | More headroom absorbs the burst while scaling | | Min replicas | **≥2** | Redundancy + reduces cold start impact | A shorter window lets the autoscaler react sooner. A longer delay prevents scale-down between waves. Lower target utilization leaves capacity available while new replicas start. ### Pre-warming for predictable bursts To pre-warm before a predictable burst (morning ramp, scheduled events): 1. Before the expected spike, bump min replicas: ```bash Request theme={"system"} curl -X PATCH \ https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/autoscaling_settings \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{"min_replica": 5}' ``` 2. After the burst subsides, reset to your normal minimum: ```bash Request theme={"system"} curl -X PATCH \ https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/autoscaling_settings \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{"min_replica": 2}' ``` > [!TIP] > > For peaks at known times, use an [autoscaling schedule](/deployment/autoscaling/schedules) to raise minimum replicas before traffic arrives. Start the schedule early enough to cover your deployment's observed cold start time. ## Scheduled traffic ### Characteristics * **Long periods of low or zero traffic**. * Large bursts tied to job schedules (hourly, daily, weekly). * Traffic patterns are predictable but infrequent. ### Common causes * ETL pipelines and data processing jobs. * Embedding backfills and batch inference. * Periodic evaluation or testing jobs. * Document processing triggered by user uploads. ### Recommended settings | Parameter | Value | Why | | - | - | - | | Min replicas | **0** (if cold starts acceptable) or **1** (during job windows) | Cost savings when idle | | Scale-down delay | **Moderate to high** | Jobs often come in waves | | Autoscaling window | **60-120s** | Don't overreact to the first few requests | | Target utilization | **70%** | Default is fine | Scale-to-zero avoids replica charges during idle periods. The moderate window filters short-lived changes at the start of a batch. If jobs come in waves, a longer delay keeps replicas warm between them. ### Scheduled pre-warming [Autoscaling schedules](/deployment/autoscaling/schedules) pre-warm replicas without an external job or API key. For a batch job that starts at the top of each hour, create an hourly window that starts at `:55` and ends after the job completes. Set the schedule's `min_replica` to the warm capacity the job needs. Outside the window, the environment returns to its default settings. Start the window earlier if replicas take more than five minutes to become ready. Use your worst observed cold start as the initial lead time, then adjust it from deployment metrics. > [accordion] External cron fallback > > If an existing orchestration system must own the schedule, run the same updates through the Management API. This example raises the minimum five minutes before an hourly job: > > ```bash Terminal theme={"system"} > 55 * * * * curl -X PATCH \ > https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/autoscaling_settings \ > -H "Authorization: Bearer $BASETEN_API_KEY" \ > -d '{"min_replica": 3}' > ``` > > This update restores scale-to-zero 30 minutes after the hour: > > ```bash Terminal theme={"system"} > 30 * * * * curl -X PATCH \ > https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/autoscaling_settings \ > -H "Authorization: Bearer $BASETEN_API_KEY" \ > -d '{"min_replica": 0}' > ``` > [!WARNING] > > If the deployment has reached zero replicas when a batch starts, Baseten must [cold-start](/deployment/autoscaling/startup-times) a replica before it can process the batch. Synchronous requests can wait while the replica starts. Setting `min_replica` to 0 allows the deployment to reach this state. For latency-sensitive batch jobs, set `min_replica` to at least 1 for the job window and start pre-warming before traffic arrives. ## Steady traffic ### Characteristics * Traffic **rises and falls gradually** over the day. * Classic diurnal pattern with no sharp edges. * Predictable, cyclical behavior. ### Common causes * Always-on inference APIs with consistent user base. * B2B applications with business-hours usage. * Production workloads with stable, mature traffic. ### Recommended settings | Parameter | Value | Why | | - | - | - | | Target utilization | **70-80%** | Uses more of each replica's capacity | | Autoscaling window | **60-120s** | Moderate reaction speed | | Scale-down delay | **300-600s** | Moderate | | Min replicas | **≥2** | Redundancy for production | Without sudden spikes, you don't need as much headroom. You can run replicas at higher utilization to reduce cost because load changes are gradual and predictable. The autoscaler has time to react. > [!TIP] > > Start with the defaults. Monitor for a week, then gradually raise target utilization while watching p95 latency. ## Next steps * [Autoscaling](/deployment/autoscaling/overview): Full parameter documentation. * [Scale a deployment](/deployment/manage/scaling): Apply these settings from the dashboard or Management API. * [Troubleshooting autoscaling](/troubleshooting/deployments#autoscaling-issues): Diagnose and fix common problems. * [Truss configuration reference](/reference/truss-configuration): Configure `predict_concurrency` in your model. # CI/CD Source: https://docs.baseten.co/deployment/ci-cd Deploy models from GitHub with the Baseten CLI or the Truss Push Action. Use the [Baseten CLI](/reference/cli/baseten/overview) to deploy and validate models or [Chains](/development/chain/deploy) from GitHub, or the [Truss Push GitHub Action](https://github.com/marketplace/actions/truss-push) when you want predict validation and automatic cleanup wrapped into a single step. Run either on pull requests to test a temporary deployment, or on merges to release a model to an environment. ## What happens during a run Depending on the deployment type and inputs, a run can include these steps: 1. **Load config**: For models, the action reads `config.yaml` from the Truss directory and extracts `model_metadata.example_model_input` for the predict step. The `predict-payload` input overrides that value. For Chains, the action detects the entrypoint class from the `.py` file. 2. **Deploy**: Pushes the model or Chain to Baseten. For a model, the action streams build logs into the GitHub Actions output. For a Chain, it prints deployment status changes and chainlet log URLs. For models, the action also names each deployment from git context: `PR-42_abc1234` for pull requests or `abc1234` for direct pushes. Override the generated name with `deployment-name`. 3. **Predict**: If the action resolves a payload that it doesn't skip, it sends a predict request and reports latency. For streaming models when the payload includes `"stream": true`, it reports time-to-first-byte, token count, and tokens per second. 4. **Cleanup**: If `cleanup: true`, deactivates the newly created deployment. Set `cleanup: false` when a model workflow deploys to an environment or when you want to inspect a deployment manually. The job summary includes the time spent waiting for the deployment to become active after the push call returns. If prediction runs, the summary also includes predict metrics. A model run includes a direct link to its deployment logs. For a Chain, use the chainlet log URLs in the deploy output. ## Prerequisites Store a Baseten API key with full access as an [encrypted secret](https://docs.github.com/en/actions/security-guides/encrypted-secrets) named `BASETEN_API_KEY` in your repository or organization settings. See [API keys](/organization/api-keys) for how to generate one. ## Deploy with the Baseten CLI The GitHub Action wraps deploy, predict validation, and cleanup in one step. If you only need to deploy, run the Baseten CLI directly in your workflow. The CLI reads your API key from the `BASETEN_API_KEY` environment variable, and `--wait` blocks the step until the deployment is active so a failed deploy fails the job: ```yaml .github/workflows/deploy.yml theme={"system"} name: Deploy to production on: push: branches: [main] jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Install the Baseten CLI run: | brew tap basetenlabs/baseten brew trust --formula basetenlabs/baseten/baseten brew install baseten - name: Push the model env: BASETEN_API_KEY: ${{ secrets.BASETEN_API_KEY }} run: baseten model push --dir ./my-model --environment production --wait ``` For the full push surface, see [`model push`](/reference/cli/baseten/model#push). To add predict validation and cleanup to a CLI-based flow, see [Validate on pull request](#validate-on-pull-request) for the Action-based equivalent. ## Deploy to an environment on merge Deploy a validated model to a specific environment every time code merges to `main`. Create `.github/workflows/deploy.yml` and add the following: ```yaml .github/workflows/deploy.yml theme={"system"} name: Deploy to production on: push: branches: [main] jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: basetenlabs/action-truss-push@v0.1.2 with: truss-directory: "./my-model" baseten-api-key: ${{ secrets.BASETEN_API_KEY }} environment: "production" cleanup: false ``` Setting `environment` publishes the model deployment to the specified environment. Setting `cleanup: false` keeps the deployment active so it can serve traffic. For a model deployed to a [regional environment](/deployment/regional-environments), add `regional-environment: true` to the action's `with` block. The action then sends its predict validation request through the regional endpoint. ## Validate on pull request Check that a deployment can answer a prediction request before releasing it. The action deploys, sends the configured payload, and deactivates the deployment after validation. For the [vLLM Qwen project](/development/model/build-your-first-model), configure `.github/workflows/validate-model.yml` with a chat-completions payload. Change the directory and payload to match your model when using another project: ```yaml .github/workflows/validate-model.yml theme={"system"} name: Validate model on: pull_request: branches: [main] jobs: validate: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: basetenlabs/action-truss-push@v0.1.2 with: truss-directory: "./qwen-2.5-3b-vllm" baseten-api-key: ${{ secrets.BASETEN_API_KEY }} predict-payload: >- {"model": "Qwen/Qwen2.5-3B-Instruct", "messages": [{"role": "user", "content": "Explain inference in one sentence."}], "max_tokens": 128} ``` The explicit `predict-payload` overrides `model_metadata.example_model_input` in `config.yaml`. Check that the job summary includes **Predict Total Time** and the deploy step reports `Predict completed in ...`. If the logs say `No predict payload configured, skipping predict check`, prediction didn't run. A successful request confirms the model accepts this input; add assertions against the response for behavior-specific regression tests. With the default (`cleanup: true`), the action deactivates the deployment after validation. If cleanup fails, [deactivate the deployment manually](/deployment/manage/lifecycle#deactivate-a-deployment). ## Deploy a Chain Deploy a Baseten Chain from a Python source file. The action auto-detects Chains when `truss-directory` points to a `.py` file: ```yaml theme={"system"} - uses: basetenlabs/action-truss-push@v0.1.2 with: truss-directory: "./chains/my_chain.py" baseten-api-key: ${{ secrets.BASETEN_API_KEY }} model-name: "my-rag-chain" cleanup: false predict-payload: '{"query": "What is Baseten?"}' ``` For Chains, provide the predict payload explicitly with `predict-payload` because there's no `config.yaml` to read example input from. > [!WARNING] > > In `v0.1.2`, the environment inputs don't promote a Chain. Leave `regional-environment` set to `false` to validate the new Chain deployment through its deployment-specific endpoint. See the [`regional-environment` input](/reference/ci/github-action#param-regional-environment) for regional validation behavior. ## Deploy multiple models Use a matrix strategy to deploy each model in your repository as a separate job. Create `.github/workflows/deploy-all.yml` and add the following: ```yaml .github/workflows/deploy-all.yml theme={"system"} name: Deploy models on: push: branches: [main] jobs: deploy: runs-on: ubuntu-latest strategy: fail-fast: false matrix: model: - path: models/text-classifier - path: models/image-generator - path: models/embeddings steps: - uses: actions/checkout@v4 - uses: basetenlabs/action-truss-push@v0.1.2 with: truss-directory: ${{ matrix.model.path }} baseten-api-key: ${{ secrets.BASETEN_API_KEY }} environment: "production" cleanup: false ``` Each matrix entry runs as a separate job. If one model fails, the others still deploy. ## Custom predict validation Override the default predict payload when your model needs a specific input shape that differs from `model_metadata.example_model_input`: ```yaml theme={"system"} - uses: basetenlabs/action-truss-push@v0.1.2 with: truss-directory: "./my-model" baseten-api-key: ${{ secrets.BASETEN_API_KEY }} predict-payload: '{"prompt": "Hello, world!", "max_new_tokens": 128}' predict-timeout: 60 ``` The action skips the predict step when the resolved JSON value is missing, `null`, `false`, `0`, an empty string (`""`), an empty object (`{}`), or an empty array (`[]`). A skipped predict step doesn't validate the deployment. ## Deploy with labels Attach metadata labels to track model deployments in your CI pipeline: ```yaml theme={"system"} - uses: basetenlabs/action-truss-push@v0.1.2 with: truss-directory: "./my-model" baseten-api-key: ${{ secrets.BASETEN_API_KEY }} labels: '{"team": "ml-platform", "triggered-by": "ci"}' ``` For more information, see [Label deployments](/deployment/manage/labels). ## Override model name Set a custom model name instead of using the name from `config.yaml`: ```yaml theme={"system"} - uses: basetenlabs/action-truss-push@v0.1.2 with: truss-directory: "./my-model" baseten-api-key: ${{ secrets.BASETEN_API_KEY }} model-name: "my-custom-name" ``` ## Use action outputs The action exposes outputs you can reference in downstream steps. This example posts the deploy time as a PR comment: ```yaml theme={"system"} steps: - uses: actions/checkout@v4 - uses: basetenlabs/action-truss-push@v0.1.2 id: deploy with: truss-directory: "./my-model" baseten-api-key: ${{ secrets.BASETEN_API_KEY }} - name: Comment on PR if: github.event_name == 'pull_request' uses: actions/github-script@v7 with: script: | github.rest.issues.createComment({ owner: context.repo.owner, repo: context.repo.repo, issue_number: context.issue.number, body: `Post-push wait to active: ${{ steps.deploy.outputs.deploy-time-seconds }} seconds. Status: ${{ steps.deploy.outputs.status }}` }) ``` See the full list of inputs and outputs in the [Truss Push GitHub Action reference](/reference/ci/github-action). ## Troubleshooting ### `deploy_timeout` The default timeout is 45 minutes, which accommodates large builds like TRT-LLM. For smaller models, reduce `deploy-timeout-minutes` to fail faster. If your model needs more time, increase the value. ### `deploy_failed` Verify the `BASETEN_API_KEY` secret first. For a model, check `config.yaml` and expand the build output in the GitHub Actions UI. For a Chain, check the source file and follow the chainlet log URLs in the deploy output. ### `predict_failed` Verify the predict payload shape matches what your model expects. Check `model_metadata.example_model_input` in `config.yaml`, or override it with `predict-payload`. For Chains, provide the predict payload explicitly. ### `cleanup_failed` The deployment may still be running. Deactivate it manually from the [Baseten dashboard](https://app.baseten.co). ### 429 too many requests The action calls Management API endpoints that are rate limited per API key. Matrix jobs that fan out across many models can exceed the per-endpoint limits. See [Management API rate limits](/reference/management-api/rate-limits) for thresholds and backoff guidance. ### No predict output The action skips prediction when the resolved JSON value is missing, `null`, `false`, `0`, an empty string (`""`), an empty object (`{}`), or an empty array (`[]`). For a model, add an example input to `config.yaml` that isn't one of these values. For a Chain, set `predict-payload` to a value that the action doesn't skip. ### `Team selection required but running in a non-interactive context` For a model workflow, your API key has access to multiple teams and Truss can't infer a single target team without a prompt. Pass the team explicitly with the `team` input on the action. If you invoke `truss push` directly, use `--team `. ```yaml theme={"system"} - uses: basetenlabs/action-truss-push@v0.1.2 with: truss-directory: "./my-model" baseten-api-key: ${{ secrets.BASETEN_API_KEY }} team: "ml-platform" ``` See [Deploy to a team](/organization/teams#use-the-truss-cli) for the team-resolution rules. Requires Truss `0.18.3` or later. The `team` input in `action-truss-push@v0.1.2` applies to models only. For a Chain workflow, use an API key that resolves to the intended team without an interactive selection. ## Next steps > [card] [Rolling deployments](/deployment/rolling-deployments) > > Promote validated deployments to production without downtime. > [card] [Environments](/deployment/environments) > > Manage staging and production environments for your models. > [card] [Manage deployments](/deployment/manage/overview) > > Scale, promote, and clean up deployments from the Baseten CLI or Management API. # Concepts Source: https://docs.baseten.co/deployment/concepts Deployments, environments, resources, autoscaling, and CI/CD on Baseten. A [deployment](/deployment/deployments) is a version of your model running on Baseten with its own API endpoint. Use an environment to give your application a stable endpoint as you release new versions. * [Deploy a model](/examples/create-a-model-with-rest) to create an endpoint and send a request. * [Call your model](/inference/calling-your-model) to connect an application to an existing deployment. * [Manage deployments](/deployment/manage/overview) to release an update, inspect logs, scale replicas, or stop compute spend. ## Deployments A deployment runs one version of your model on a selected instance type. Each `baseten model push` creates a deployment. Multiple deployments of the same model can run at once, allowing you to test a new version without changing production traffic. Deactivate a deployment to stop serving and billing, or delete it when you no longer need it. For Python Truss models, [use the development loop](/development/model/deploy-and-iterate) to create a **development deployment**, a mutable instance that live-reloads as you edit your model code. Promoting a development deployment to an environment creates a published deployment. For custom Docker servers and engine configurations, [publish and validate an update](/development/model/deploy-and-iterate#publish-and-validate-an-update) before promotion. ![Baseten dashboard showing multiple model deployments](https://mintcdn.com/baseten-preview/W3NbEem9OZkF5rdB/images/deployments.png?fit=max\&auto=format\&n=W3NbEem9OZkF5rdB\&q=85\&s=833c7f33c3a39b50502326b9cec14090) ### Model lab deployments For deployment restrictions on models distributed by model labs, see [Customer availability and artifact access](/labs/platform#customer-availability-and-artifact-access). ## Environments [Environments](/deployment/environments) provide stable endpoints that persist as you release new deployments. A typical setup uses one environment for testing and another for production traffic. Each environment has its own endpoint, autoscaling settings, and metrics. Promoting a deployment moves the environment's traffic to that version without changing the URL called by your application. ![Deployment environments with development and production endpoints](https://mintcdn.com/baseten-preview/W3NbEem9OZkF5rdB/images/deployment-environments.png?fit=max\&auto=format\&n=W3NbEem9OZkF5rdB\&q=85\&s=e922f3e12c24577d6594ca58f80431ee) ## Resources Every deployment runs on an [instance type](/deployment/resources) that defines its GPU, CPU, and memory. Set the instance type in `config.yaml`, or change it for a published deployment in the Baseten dashboard. Select an instance based on the model's memory requirements, latency target, and expected traffic. ![Resource configuration showing GPU instance type selection](https://mintcdn.com/baseten-preview/W3NbEem9OZkF5rdB/images/deployment-resources.png?fit=max\&auto=format\&n=W3NbEem9OZkF5rdB\&q=85\&s=028f7b8a7c5e1d92f0b55f2eec8aad11) ## Autoscaling [Autoscaling](/deployment/autoscaling/overview) adjusts the number of replicas as request load changes. Configure the minimum and maximum replicas, concurrency target, and scale-down delay. A deployment can scale to zero when idle, so a synchronous request can wait for a replica to start. See [Startup times](/deployment/autoscaling/startup-times) for ways to reduce startup time. ![Autoscaling configuration with replica count and concurrency settings](https://mintcdn.com/baseten-preview/W3NbEem9OZkF5rdB/images/deployment-autoscaling.png?fit=max\&auto=format\&n=W3NbEem9OZkF5rdB\&q=85\&s=f867eb7ccef178bae2fd11117365bebb) For the mechanics of how the autoscaler tracks concurrent requests and adjusts replicas, see [How Baseten works](/concepts/howbasetenworks#autoscaling). For engine-specific autoscaling settings (BEI and Engine-Builder-LLM), see [Autoscaling engines](/engines/performance-concepts/autoscaling-engines). ## Request lifecycle Before your model code runs, each request passes through authentication, routing, and replica selection. See [Request lifecycle](/deployment/autoscaling/request-lifecycle) to understand queuing, load shedding, timeouts, and HTTP status codes. ## CI/CD Use the [Truss Push GitHub Action](/deployment/ci-cd) to deploy a model from a Git repository, validate it with a predict request, and optionally promote it to production. Configure the workflow to run for selected branches, pushes, or pull requests. # Deployments Source: https://docs.baseten.co/deployment/deployments Understand published and development deployments, how they relate to environments, and which lifecycle action to use. A deployment is one version of a model running on Baseten. It has its own ID, model artifact, instance type, autoscaling settings, replicas, and API endpoint. A model can have multiple deployments at the same time. Use this page to understand the deployment resource and its relationship to environments. To perform an operation, see [Manage deployments](/deployment/manage/overview). To send an inference request, see [Call a model](/inference/calling-your-model). ## Deployment types ### Published deployments By default, `baseten model push` creates a published deployment. Its model artifact doesn't live reload, so a code or configuration change requires another push. Published model deployments support the full set of autoscaling and environment promotion settings. A published deployment can serve requests through its deployment-specific endpoint without belonging to an environment. ### Development deployment Use `baseten model push --watch` for a model or `baseten truss chains push --watch` for a Chain to create a development deployment. Development deployments support a fast edit-and-reload loop: * Source changes live reload without another deployment. * The deployment uses one replica when active and scales to zero when idle. * The deployment can't scale beyond one replica or use zero-downtime updates. Promoting a development deployment to an environment creates a new published deployment with its own ID and starts an image build. After the deployment becomes ready, Baseten runs the environment's configured promotion workflow. The promotion completes when the environment points to the new published deployment. ## Environments and promotion An [environment](/deployment/environments) gives a model a stable endpoint and a reusable set of resource, autoscaling, and promotion settings. The environment points to the deployment that currently serves its traffic. * A deployment doesn't need an environment to serve requests. * The production environment exists by default. You can add environments such as staging. * Promoting a deployment changes which version serves the environment endpoint without changing that endpoint's URL. * Promotion can reuse the source deployment or create a new published deployment, depending on the source type and the environment settings. ### Rolling deployments Rolling deployment is an environment promotion strategy, not a deployment type. It incrementally provisions the candidate deployment, shifts traffic, and removes the previous deployment's replicas. You can pause, resume, cancel, or force-complete the promotion. For more information, see [Rolling deployments](/deployment/rolling-deployments). ### Canary deployments (deprecated) > [!WARNING] > > Canary promotions are deprecated for models, and you can no longer turn them on. Use [rolling deployments](/deployment/rolling-deployments) for incremental traffic shifting. The management API rejects requests that set `ramp_up_while_promoting` to `true` on a model environment. Enabling rolling deployments on a model environment turns canary promotions off. Chains still support canary promotions. ## Deployment metadata Baseten addresses a deployment by its model ID and deployment ID. Names and labels help people and automation identify deployments. A deployment name is a human-readable identifier and doesn't change the deployment's API path. Set it with [`baseten model push --deployment-name`](/reference/cli/baseten/model#push), or rename it later with the Baseten CLI or in the Baseten dashboard: ```bash Command theme={"system"} baseten model deployment rename --model-id --deployment-id --new-name ``` ```txt Output theme={"system"} Renamed deployment to ``` For more information, see [`deployment rename`](/reference/cli/baseten/model#rename-1). [Deployment labels](/deployment/manage/labels) are JSON key-value metadata for ownership, source control, or CI information. The dashboard shows a deployment's effective [regional placement](/deployment/regional-deployments#how-placement-works). It shows the selected deployment region first, then any regions allowed by the compliance policy. If neither constraint applies, it shows **Global**. You can't change a published deployment's labels after creation. A later development push that includes labels replaces the development deployment's labels. A push that omits labels preserves them. ## Deployment lifecycle Use [Manage deployments](/deployment/manage/overview) to promote, scale, inspect, deactivate, activate, or delete a deployment. The task guides explain each operation and its effect on inference traffic. ## Next steps > [card] [Manage deployments](/deployment/manage/overview) > > Scale, promote, inspect, deactivate, and delete deployments. > [card] [Environments](/deployment/environments) > > Configure stable endpoints and promotion behavior. > [card] [Autoscaling](/deployment/autoscaling/overview) > > Configure replica capacity and scaling behavior. > [card] [Call a model](/inference/calling-your-model) > > Send inference requests to a deployment or environment. # Environments Source: https://docs.baseten.co/deployment/environments Manage your model's release cycles with environments. An environment is a named release target for a model, such as production or staging. It has a stable endpoint and its own resource, autoscaling, monitoring, and promotion settings. Promote a new deployment to update the version that serves the environment without changing the endpoint your client calls. ![Baseten dashboard showing model environments and their deployments](https://mintcdn.com/baseten-preview/W3NbEem9OZkF5rdB/images/deployment-environments.png?fit=max\&auto=format\&n=W3NbEem9OZkF5rdB\&q=85\&s=e922f3e12c24577d6594ca58f80431ee) Use a separate environment to validate a deployment without changing production traffic. ## Deployment management Use environments to validate and release deployments with: * Automated tests and evaluations. * Manual testing in pre-production. * Performance tests with copies of live requests through [shadow traffic](#shadow-traffic). * Gradual traffic shifts with [rolling deployments](/deployment/rolling-deployments). > [!NOTE] > > Baseten enables rolling deployments by default for new model environments. > Existing environments keep their promotion settings. To change this setting, [enable rolling deployments](/deployment/rolling-deployments#enable-rolling-deployments) or [turn off rolling deployments](/deployment/rolling-deployments#disable-rolling-deployments) for the environment. When you promote a deployment, Baseten applies the environment's scaling and monitoring settings: * Dedicated API endpoint. For more information, see [Predict endpoints](/reference/inference-api/overview#predict-endpoints). * Autoscaling controls: Scale behavior is managed per environment. * Monitoring and metrics: scope [logs](/observability/logs#scope-by-environment-or-deployment) and [metrics](/observability/metrics) to the environment in the dashboard, or [export environment metrics](/observability/export-metrics/overview) to your own observability stack. The production environment operates like any other environment but has restrictions: * It can't be deleted unless the entire model is removed. * You can't create additional environments named "production." ## Shadow traffic Shadow traffic copies a share of an environment's live HTTP requests to another environment or deployment in the same model, so you can test a deployment on real traffic before you promote it. Clients continue to receive responses from the source deployment, while Baseten discards responses from the target deployment. Shadowing is best effort. Depending on platform and target conditions, Baseten might not copy every sampled request, and copies can fail. Skipped and failed copies don't affect the client's response, but the target can receive a smaller share of requests than the percentage you set. The target must meet the source's regional and compliance constraints. A target environment receives the copies on whichever deployment it's serving. > [!NOTE] > > Shadow traffic is pre-release. The dashboard shows the **Traffic routing** > section only for workspaces with traffic routing rules enabled, and the > `baseten model environment traffic-rule` commands may change their arguments, > flags, and output. > [tab] UI > > **To start shadow traffic**: > > 1. Sign in to your workspace at > [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the > sidebar, then select your model. > 2. Under **Environments**, select the source environment. > 3. In the **Traffic routing** section, choose **Add rule**. > 4. For **Target**, select the environment or deployment to receive the copies. > 5. For **Traffic to shadow**, set the percentage of requests to copy. > Shadowing is best effort, so the target can receive a smaller share than > configured. > 6. For **Expiration**, choose how long the rule runs, up to 7 days. > 7. For **Sampling**, choose **Random** to pick each request independently, > or **Session-aware** to keep a session's requests together. > 8. To cancel a copy when the client cancels the original request, select > **Propagate client cancellation to the target**. > 9. Choose **Start shadowing**. > > The **Traffic routing** section shows the running rule. > > **To change a shadow rule**: > > 1. In the source environment's **Traffic routing** section, choose **Edit**. > 2. Change the percentage, expiration, sampling, or cancellation, then choose > **Save**. > > **To stop shadow traffic**: > > 1. In the source environment's **Traffic routing** section, choose **Stop**. > 2. Choose **Stop rule** to confirm. > [tab] Baseten CLI > > **To start shadow traffic**: > > 1. Create a rule on the source environment. Replace `` with an > ISO 8601 time, such as `2026-11-10T18:00:00Z`, at most 7 days out. > > Copy 25% of production's requests to the deployment serving staging, > picking each request independently: > > ```bash theme={"system"} > baseten model environment traffic-rule create \ > --model-id abc123 \ > --environment production \ > --type shadow \ > --target-environment staging \ > --percent-traffic 25 \ > --expires-at \ > --sampling-mode random \ > --propagate-client-cancellation true > ``` > > Copy 10% of production's requests to one deployment, keeping each > session's requests together: > > ```bash theme={"system"} > baseten model environment traffic-rule create \ > --model-id abc123 \ > --environment production \ > --type shadow \ > --target-deployment-id def456 \ > --percent-traffic 10 \ > --expires-at \ > --sampling-mode user-session \ > --propagate-client-cancellation false > ``` > > 2. List the environment's rules to confirm the rule and get its ID: > > ```bash theme={"system"} > baseten model environment traffic-rule list --model-id abc123 --environment production > ``` > > **To change a shadow rule**: > > Run the following command with the rule's ID. Only the flags you pass > change: > > ```bash theme={"system"} > baseten model environment traffic-rule update \ > --model-id abc123 \ > --environment production \ > --rule-id \ > --percent-traffic 50 > ``` > > **To stop shadow traffic**: > > Run the following command with the rule's ID. The command asks for > confirmation; pass `--yes` to skip it when scripting: > > ```bash theme={"system"} > baseten model environment traffic-rule delete --model-id abc123 --environment production --rule-id > ``` > [tab] Management API > > **To start shadow traffic**: > > Add a rule to the source environment's `traffic_routing_settings`. Replace > `` with an ISO 8601 time with a timezone, at most 7 days out: > > ```bash theme={"system"} > curl -X PATCH https://api.baseten.co/v1/models/{model_id}/environments/production \ > -H "Authorization: Bearer $BASETEN_API_KEY" \ > -H "Content-Type: application/json" \ > -d '{ > "traffic_routing_settings": { > "rules": [ > { > "type": "SHADOW", > "targets": [{"environment": "staging"}], > "percent_traffic": 25, > "expires_at": "", > "sampling_mode": "RANDOM", > "propagate_client_cancellation": true > } > ] > } > }' > ``` > > To target one deployment, set `targets` to `[{"deployment_id": "def456"}]`. > To keep each session's requests together, set `sampling_mode` to > `USER_SESSION`. The response's `environment.traffic_routing.rules` lists > the running rule with its `id`. > > **To change a shadow rule**: > > Send the complete rule with its `id`. The API replaces the rule, so include > every field; rules you omit stay unchanged. To change the target, delete the > rule and create a new one. > > ```bash theme={"system"} > curl -X PATCH https://api.baseten.co/v1/models/{model_id}/environments/production \ > -H "Authorization: Bearer $BASETEN_API_KEY" \ > -H "Content-Type: application/json" \ > -d '{ > "traffic_routing_settings": { > "rules": [ > { > "id": "", > "type": "SHADOW", > "targets": [{"environment": "staging"}], > "percent_traffic": 50, > "expires_at": "", > "sampling_mode": "RANDOM", > "propagate_client_cancellation": true > } > ] > } > }' > ``` > > **To stop shadow traffic**: > > Pass the rule's ID in `delete_rules`: > > ```bash theme={"system"} > curl -X PATCH https://api.baseten.co/v1/models/{model_id}/environments/production \ > -H "Authorization: Bearer $BASETEN_API_KEY" \ > -H "Content-Type: application/json" \ > -d '{"traffic_routing_settings": {"delete_rules": [""]}}' > ``` Session-aware sampling groups requests by the session key the client sends, such as a consistent `x-session-affinity` header value, so Baseten copies every request in a sampled session. Compare the source's and target's [logs and metrics](/deployment/manage/logs-and-metrics) over the same time range. Target metrics include any traffic sent directly to the target, and the target's replica-hours bill separately. A rule also ends when it expires, when a promotion goes into its source or target environment or promotes its target deployment, or when you deactivate or delete its source or target. A source environment has one active shadow rule at a time, so creating another fails until the rule expires or you delete it. Stopping a rule leaves both deployments running. For more information, see [`baseten model environment traffic-rule`](/reference/cli/baseten/model#environment-traffic-rule) and [Update an environment's settings](/reference/management-api/environments/update-an-environments-settings). ## Manage environments from the Baseten CLI The [`baseten model environment`](/reference/cli/baseten/model) commands work on the same environments as the dashboard and the Management API: * `baseten model environment list --model-id `: list a model's environments. * `baseten model environment describe --model-id --environment production`: inspect an environment's current deployment, autoscaling, and promotion settings. * `baseten model environment update-autoscaling` and `update-promotion`: change an environment's settings; see [schedule autoscaling](/deployment/autoscaling/schedules) and [rolling deployments](/deployment/rolling-deployments). * `baseten model deployment promote --model-id --deployment-id `: promote a deployment into an environment; see [Manage deployments](/deployment/manage/overview). ## Custom environments Each workspace has a configured per-model environment limit. In addition to the standard production environment, you can create custom environments until the model reaches that limit. Create them from the model management page in the Baseten dashboard or through the [create environment endpoint](/reference/management-api/environments/create-an-environment) in the Management API. ## Deployment promotion When you promote a deployment to an environment, Baseten associates the deployment with that environment and applies the environment's autoscaling settings. If Baseten can reuse the deployment directly, promotion doesn't create a new deployment ID. Otherwise, Baseten creates a deployment with a unique ID, initializes its resources, and replaces the existing deployment in that environment. A new deployment is created when: * You promote a development deployment. * The deployment is already associated with another environment. * The environment has a different instance type or resource profile. * The deployment moves to a different [region](/deployment/regional-deployments). * The applicable [compliance policy](/observability/security#view-your-compliance-policy) differs from the deployment's current policy. * [Re-deploy on promotion](#re-deploy-on-promotion) is enabled. When Baseten creates a new deployment, it applies the environment's autoscaling settings to that deployment. After promotion completes, the environment's [promotion cleanup strategy](/deployment/rolling-deployments#deployment-cleanup) controls the deployment it replaced. If Baseten creates a new deployment from a published source, the new deployment reuses the source image. Promoting a development deployment creates a new published deployment from its current state and triggers an image build. ### Published deployment promotion If Baseten can reuse a published deployment, it updates that deployment's autoscaling settings to match the environment. If promotion creates a new deployment, the source published deployment keeps its existing settings. ## Direct deployment to an environment Run the following command from your Truss project to deploy to an existing environment. Replace `` with its name: ```sh Terminal theme={"system"} baseten model push --environment --wait ``` To create a new custom environment during the push, run the following command: ```sh Terminal theme={"system"} baseten model push --environment staging --create-environment-if-missing --wait ``` Without `--create-environment-if-missing`, a push to a missing custom environment fails. Production already exists for every model. `--wait` confirms deployment readiness. Before sending application traffic, [verify that the environment points to the promoted deployment](/deployment/manage/lifecycle#promote-to-an-environment) and send a request to its stable endpoint. The Truss CLI equivalent for an existing environment is `truss push --environment `. > [!NOTE] > > Only one active promotion per environment is allowed at a time. ## Environment access in code Python Truss models receive the environment through the `environment` keyword argument. Its value is `None` when the deployment isn't associated with an environment, so normalize it before reading its name: ```python model/model.py theme={"system"} def __init__(self, **kwargs): self._environment = kwargs.get("environment") or {} ``` You can use the environment in your `load()` method to configure per-environment behavior: ```python model/model.py theme={"system"} def load(self): if self._environment.get("name") == "production": self.setup_sentry() self.model = self.load_production_weights() else: self.model = self.load_default_weights() ``` Validate an unassociated deployment first: it should load the default weights. After promotion to production, confirm that initialization uses the production weights through your model's prediction result or [deployment logs](/deployment/manage/logs-and-metrics#fetch-and-stream-logs). If you use environment-specific configuration in `load()`, you'll need to enable re-deploy on promotion to ensure the environment is correctly initialized after each promotion. See [Re-deploy on promotion](#re-deploy-on-promotion) for details. > [!NOTE] > > The `environment` keyword argument is only available to Python Truss models. Custom servers read the environment name from the filesystem instead. See [Environment name](/development/model/custom-server#environment-name). ## Re-deploy on promotion By default, promoting a deployment reuses the existing deployment when possible. This is the fastest promotion path, but it means `load()` doesn't re-run. Any environment-specific configuration set during the original `load()` call persists, even if the deployment moves to a different environment. You can configure an environment to create a fresh deployment every time you promote to it. For a published source deployment, the new deployment reuses the source image and reruns `load()` with the target environment's context, so environment-specific configuration takes effect. Enable this if your `load()` method uses `kwargs["environment"]` to configure per-environment behavior, or if you promote the same source deployment to multiple environments and want each to get a fresh deployment. To enable re-deploy on promotion: 1. Sign in to your workspace at [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the sidebar, then select your model. 2. Open the environment's settings. 3. Toggle **Re-deploy when promoting**. You can also set it through the [update environment settings endpoint](/reference/management-api/environments/update-an-environments-settings). > [!NOTE] > > If you promote a deployment that's already associated with an environment, Baseten creates a new deployment regardless of this setting. ## Regional environments Regional environments constrain replicas to a designated geographic region and provide a dedicated endpoint for region-specific inference routing. Baseten configures the regional restriction for an environment name within a team. See [Regional environments](/deployment/regional-environments) for setup, required-endpoint behavior, endpoint formats, and request restrictions. To select a region for an individual deployment instead, see [regional deployments](/deployment/regional-deployments). ## Environment deletion You can delete environments, except for production. To remove a production deployment, first promote another deployment to production or delete the entire model. * Deleted environments are removed from the overview but remain in billing history. * They don't consume resources after deletion. * API requests to a deleted environment return a 404 error. # Label deployments Source: https://docs.baseten.co/deployment/manage/labels Add labels when you push a deployment, then view them in the dashboard, the Baseten CLI, or the Management API. Labels are a JSON object stored with a deployment. Use them to record metadata such as the owning team, source commit, or CI run. Published deployment labels remain fixed after creation. To change them, create another deployment with new labels. For a development deployment, passing labels on a later push replaces its current labels. Omitting labels preserves them. A label object can be up to 2 KB after JSON serialization. Label keys and string values can contain printable ASCII characters except spaces. ## Set labels > [tab] Baseten CLI > > **To label a deployment**: > > ```bash theme={"system"} > baseten model push --labels '{"team": "ml-platform", "env": "staging"}' > ``` > > For more information, see [`baseten model push`](/reference/cli/baseten/model). > [tab] Truss CLI > > **To label a deployment**: > > ```bash theme={"system"} > truss push --labels '{"team": "ml-platform", "env": "staging"}' > ``` > > For more information, see [`truss push`](/reference/cli/truss/push). > [tab] Python SDK > > **To label a deployment**: > > ```python theme={"system"} > import truss > > truss.push("./my-model", labels={"team": "ml-platform", "env": "staging"}) > ``` > > For more information, see [`truss.push()`](/reference/sdk/truss/push). > [tab] Management API > > **To label a deployment**: > > Add `labels` to the `deployment` object in both the prepare and create > requests: > > ```json theme={"system"} > { > "labels": { > "team": "ml-platform", > "env": "staging" > } > } > ``` > > For the complete prepare, upload, and create sequence, see > [Deploy with the Management API](/examples/create-a-model-with-rest#management-api). > [tab] CI/CD > > **To label a deployment**: > > ```yaml theme={"system"} > - uses: basetenlabs/action-truss-push@v0.1.2 > with: > truss-directory: "./my-model" > baseten-api-key: ${{ secrets.BASETEN_API_KEY }} > labels: '{"team": "ml-platform", "triggered-by": "ci"}' > ``` > > For more information, see the > [deploy GitHub Action](/reference/ci/github-action) and > [Deploy with labels](/deployment/ci-cd#deploy-with-labels). ## View labels After you push the deployment, view its labels in the dashboard, the Baseten CLI, or the Management API: > [tab] UI > > **To view a deployment's labels**: > > 1. Sign in to your workspace at > [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the > sidebar. > 2. Select your model. > 3. Select the deployment under **Deployments**. > 4. Find **Labels** in the deployment details, below **Truss config**. > [tab] Baseten CLI > > **To view a deployment's labels**: > > JSON output includes labels. The text summary omits them. > > Print the whole deployment record with `--output json`: > > ```bash Command theme={"system"} > baseten model deployment describe --model-id abc123 --deployment-id def456 --output json > ``` > > ```json Output theme={"system"} > { > "id": "def456", > "name": "deployment-1", > "model_id": "abc123", > "status": "ACTIVE", > "instance_type_name": "1x2 - 1 vCPU, 2 GiB RAM", > "labels": { > "env": "staging", > "team": "ml-platform" > }, > ... > } > ``` > > Print just the labels with `--jq`: > > ```bash Command theme={"system"} > baseten model deployment describe --model-id abc123 --deployment-id def456 --jq '.labels' > ``` > > ```json Output theme={"system"} > { > "env": "staging", > "team": "ml-platform" > } > ``` > > For more information, see > [`baseten model deployment describe`](/reference/cli/baseten/model). > [tab] Management API > > **To view a deployment's labels**: > > ```bash Request theme={"system"} > curl "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}" \ > -H "Authorization: Bearer $BASETEN_API_KEY" > ``` > > ```json Response theme={"system"} > { > "id": "def456", > "name": "deployment-1", > "model_id": "abc123", > "status": "ACTIVE", > "labels": { > "team": "ml-platform", > "env": "staging" > }, > ... > } > ``` > > For more information, see > [Get a model's deployment by ID](/reference/management-api/deployments/gets-a-models-deployment-by-id). ## Next steps Use labels in CI/CD to compare the deployment running on Baseten with the commit or workflow run that created it. * [Manage the deployment lifecycle](/deployment/manage/lifecycle) to promote or retire a labeled deployment. * [CI/CD](/deployment/ci-cd) to attach labels automatically from GitHub Actions. * [Deployments](/deployment/deployments) for how deployments, environments, and promotion fit together. # Manage the deployment lifecycle Source: https://docs.baseten.co/deployment/manage/lifecycle Promote, deactivate, activate, and delete deployments from the dashboard, Baseten CLI, or Management API. Manage your deployment through its lifecycle. After you push a model, promote its deployment to serve an environment, deactivate it to stop compute spend, activate it to bring it back, and delete it when you no longer need it. For what each deployment state means, see [Deployments](/deployment/deployments). ## Promote to an environment Promote a validated deployment to production (or a custom environment) to route that environment's traffic to it. What happens to the previously promoted deployment is controlled by the target environment's promotion cleanup strategy; see [Environments](/deployment/environments) for the concepts and [Rolling deployments](/deployment/rolling-deployments) for incremental traffic shifting. Promotion can create a new published deployment with a different ID. For example, Baseten creates one when you promote a development deployment. A promotion can also reuse the source deployment. In both cases, use the `id` in the promotion response as the promoted deployment ID. The request can return while that deployment is still `BUILDING`, `DEPLOYING`, or `UPDATING`, before the target environment points to it. The CLI and cURL examples use [`jq`](https://jqlang.github.io/jq/) to extract fields from JSON responses. The promotion is complete when the target environment's `current_deployment.id` matches the returned `promoted_deployment_id`. Check the environment rather than the source deployment, then test the stable endpoint with your model's prediction input. > [tab] UI > > **To promote a deployment**: > > 1. Sign in to your workspace at > [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the > sidebar. > 2. Select your model. > 3. Select the deployment under **Deployments**. > 4. Choose **Promote to...**. > 5. In the **Promote deployment** dialog, choose the target environment. > 6. Choose **Promote** to confirm. > > **To verify the promotion**: > > On the model's overview, confirm the environment card's > **Current deployment** shows the promoted deployment. > [tab] Baseten CLI > > **To promote a deployment**: > > Promote to production, the CLI's default target, and save the returned > deployment ID: > > ```bash Command theme={"system"} > target_environment=production > promoted_deployment_id="$( > baseten model deployment promote \ > --model-id \ > --deployment-id \ > --output json | jq -r '.id' > )" > printf 'Promoted deployment ID: %s\n' "$promoted_deployment_id" > ``` > > ```txt Output theme={"system"} > Promoted deployment ID: > ``` > > Promote to another existing environment and save the returned deployment ID: > > ```bash Command theme={"system"} > target_environment=staging > promoted_deployment_id="$( > baseten model deployment promote \ > --model-id \ > --deployment-id \ > --environment "$target_environment" \ > --output json | jq -r '.id' > )" > printf 'Promoted deployment ID: %s\n' "$promoted_deployment_id" > ``` > > ```txt Output theme={"system"} > Promoted deployment ID: > ``` > > The CLI prompts for confirmation. Pass `--yes` to skip it for > non-interactive automation, where the flag is required. > > **To verify the promotion**: > > Run this check again while the promotion is in progress: > > ```bash Command theme={"system"} > current_deployment_id="$( > baseten model environment describe \ > --model-id \ > --environment "$target_environment" \ > --output json | jq -r '.current_deployment.id // empty' > )" > > if [ "$current_deployment_id" = "$promoted_deployment_id" ]; then > printf 'Promotion complete: %s\n' "$current_deployment_id" > else > printf 'Promotion not complete\n' > fi > ``` > > ```txt Output theme={"system"} > Promotion complete: > ``` > [tab] Management API > > **To promote a deployment**: > > The response examples below show development deployment promotions. The > returned published deployment is still `BUILDING`, so the target > environment doesn't point to it yet. A response for a reused published > deployment can show `ACTIVE` or `UPDATING` and include the target > environment's name. > > Promote to production and save the returned deployment ID: > > ```bash Request theme={"system"} > target_environment=production > promotion_response="$( > curl -sS -X POST "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/promote" \ > -H "Authorization: Bearer $BASETEN_API_KEY" > )" > promoted_deployment_id="$(printf '%s' "$promotion_response" | jq -r '.id')" > printf '%s\n' "$promotion_response" | > jq '{id, is_production, is_development, status, environment}' > ``` > > ```json Response theme={"system"} > { > "id": "", > "is_production": false, > "is_development": false, > "status": "BUILDING", > "environment": null > } > ``` > > Promote to another environment by sending the source deployment ID to that > environment's promote endpoint. Save the ID from the response: > > ```bash Request theme={"system"} > target_environment=staging > promotion_response="$( > curl -sS -X POST "https://api.baseten.co/v1/models/{model_id}/environments/$target_environment/promote" \ > -H "Authorization: Bearer $BASETEN_API_KEY" \ > -H "Content-Type: application/json" \ > -d '{"deployment_id": "{deployment_id}"}' > )" > promoted_deployment_id="$(printf '%s' "$promotion_response" | jq -r '.id')" > printf '%s\n' "$promotion_response" | > jq '{id, is_production, is_development, status, environment}' > ``` > > ```json Response theme={"system"} > { > "id": "", > "is_production": false, > "is_development": false, > "status": "BUILDING", > "environment": null > } > ``` > > **To verify the promotion**: > > Run this check again while the promotion is in progress: > > ```bash Request theme={"system"} > environment_response="$( > curl -sS "https://api.baseten.co/v1/models/{model_id}/environments/$target_environment" \ > -H "Authorization: Bearer $BASETEN_API_KEY" > )" > current_deployment_id="$( > printf '%s' "$environment_response" | > jq -r '.current_deployment.id // empty' > )" > > if [ "$current_deployment_id" = "$promoted_deployment_id" ]; then > printf 'Promotion complete: %s\n' "$current_deployment_id" > else > printf 'Promotion not complete\n' > fi > ``` > > ```txt Output theme={"system"} > Promotion complete: > ``` > [tab] CI/CD > > **To promote from CI/CD**: > > Each push to `main` deploys the Truss and promotes the new deployment to > production: > > ```yaml .github/workflows/deploy.yml theme={"system"} > name: Deploy to production > > on: > push: > branches: [main] > > jobs: > deploy: > runs-on: ubuntu-latest > steps: > - uses: actions/checkout@v4 > > - uses: basetenlabs/action-truss-push@v0.1.2 > id: deploy > with: > truss-directory: "./my-model" > baseten-api-key: ${{ secrets.BASETEN_API_KEY }} > environment: "production" > cleanup: false > > - name: Verify production promotion > env: > BASETEN_API_KEY: ${{ secrets.BASETEN_API_KEY }} > MODEL_ID: ${{ steps.deploy.outputs.model-id }} > PROMOTED_DEPLOYMENT_ID: ${{ steps.deploy.outputs.deployment-id }} > run: | > for attempt in $(seq 1 60); do > current_deployment_id="$( > curl --fail --silent --show-error \ > "https://api.baseten.co/v1/models/$MODEL_ID/environments/production" \ > -H "Authorization: Bearer $BASETEN_API_KEY" | > jq -r '.current_deployment.id // empty' > )" > if [ "$current_deployment_id" = "$PROMOTED_DEPLOYMENT_ID" ]; then > printf 'Promotion complete: %s\n' "$current_deployment_id" > exit 0 > fi > sleep 5 > done > printf 'Production did not switch to the promoted deployment\n' >&2 > exit 1 > ``` > > The verification step succeeds when production points to this deployment. > If it doesn't switch within 5 minutes after readiness, the step fails. > Add a model-specific prediction check against the stable environment > endpoint before treating the release as complete; see > [Validate on pull request](/deployment/ci-cd#validate-on-pull-request) > for prediction payload configuration. > > For more information, see [CI/CD](/deployment/ci-cd) and the > [Truss Push GitHub Action reference](/reference/ci/github-action). For more information, see the [promote deployment](/reference/management-api/deployments/promote/promotes-a-deployment-to-production) and [promote to environment](/reference/management-api/deployments/promote/promotes-a-deployment-to-an-environment) endpoints. ## Deactivate a deployment Deactivate a deployment to stop compute spend without deleting it. The deployment keeps its configuration and stays visible in the dashboard, but releases its replicas. Requests to a deactivated deployment fail with a `400` error; see [troubleshooting](/troubleshooting/deployments#issue-requests-fail-with-model-version-is-deactivated) for the error and recovery. > [tab] UI > > **To deactivate a deployment**: > > 1. Sign in to your workspace at > [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the > sidebar. > 2. Select your model. > 3. Select the deployment under **Deployments**. > 4. Choose **Deactivate deployment**. > 5. Choose **Yes, deactivate** to confirm. > [tab] Baseten CLI > > **To deactivate a deployment**: > > ```bash Command theme={"system"} > baseten model deployment deactivate --model-id --deployment-id > ``` > > ```txt Output theme={"system"} > Deactivated deployment > ``` > > The CLI prompts for confirmation; pass `--yes` to skip it (required when > scripting). > [tab] Management API > > **To deactivate a deployment**: > > ```bash Request theme={"system"} > curl -X POST "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/deactivate" \ > -H "Authorization: Bearer $BASETEN_API_KEY" > ``` > > ```json Response theme={"system"} > {"success": true} > ``` The deployment's `status` moves to `INACTIVE` within a few seconds. If you want the deployment to keep serving but stop paying for idle replicas, [scale to zero](/deployment/manage/scaling#scale-to-zero) instead. Deactivating also releases the deployment's slot against your workspace's [deployment limit](/troubleshooting/deployments#issue-youve-reached-the-maximum-number-of-deployed-models). Scaling to zero doesn't, since the endpoint stays live. ## Activate a deployment Activate an inactive deployment to bring it back. Activation redeploys the model, so the deployment passes through `DEPLOYING` before reaching `ACTIVE`. > [tab] UI > > **To activate a deployment**: > > 1. Sign in to your workspace at > [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the > sidebar. > 2. Select your model. > 3. Select the inactive deployment under **Deployments**. > 4. Choose **Activate deployment**. > [tab] Baseten CLI > > **To activate a deployment**: > > ```bash Command theme={"system"} > baseten model deployment activate --model-id --deployment-id > ``` > > ```txt Output theme={"system"} > Activated deployment > ``` > [tab] Management API > > **To activate a deployment**: > > ```bash Request theme={"system"} > curl -X POST "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/activate" \ > -H "Authorization: Bearer $BASETEN_API_KEY" > ``` > > ```json Response theme={"system"} > {"success": true} > ``` For more information, see [`activate`](/reference/cli/baseten/model#activate) and the [activate endpoint](/reference/management-api/deployments/activate/activates-a-deployment). ## Delete a deployment Delete deployments to clean up finished experiments and stale versions. > [!WARNING] > > Deletion is irreversible, and requests to a deleted deployment return `404`. > [Deactivate](#deactivate-a-deployment) instead if you might need the > deployment again. A deployment that's associated with an environment, or is the only deployment of a model, can't be deleted; [push a new deployment](/reference/cli/truss/push) and promote it first, or delete the whole model. > [tab] UI > > **To delete a deployment**: > > 1. Sign in to your workspace at > [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the > sidebar. > 2. Select your model. > 3. Select the deployment under **Deployments**. > 4. Choose **Delete deployment** and confirm. > > The button is disabled while the deployment serves an environment or is > the model's only deployment. > [tab] Baseten CLI > > **To delete a deployment**: > > ```bash Command theme={"system"} > baseten model deployment delete --model-id --deployment-id > ``` > > ```txt Output theme={"system"} > Deleted deployment > ``` > [tab] Management API > > **To delete a deployment**: > > ```bash Request theme={"system"} > curl -X DELETE "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}" \ > -H "Authorization: Bearer $BASETEN_API_KEY" > ``` > > ```json Response theme={"system"} > {"id": "{deployment_id}", "deleted": true, "model_id": "{model_id}"} > ``` Deleting a model removes all of its deployments. See [`baseten model delete`](/reference/cli/baseten/model#delete) or the [delete model endpoint](/reference/management-api/models/deletes-a-model-by-id). For more information, see the Management API [delete deployment endpoint](/reference/management-api/deployments/deletes-a-models-deployment-by-id) and the Baseten CLI [`deployment delete`](/reference/cli/baseten/model#delete-1) reference. ## Next steps * [Scale a deployment](/deployment/manage/scaling) to change replica behavior without a lifecycle change. * [Pull logs and metrics](/deployment/manage/logs-and-metrics) to verify a deployment's health after a transition. * [CI/CD](/deployment/ci-cd) to run deploys and promotions from GitHub Actions. # Pull logs and metrics Source: https://docs.baseten.co/deployment/manage/logs-and-metrics View logs and metrics in the dashboard, or fetch them with the Baseten CLI or Management API. Use the dashboard, Baseten CLI, or Management API to investigate a deployment or monitor an environment. For dashboard filtering and request correlation, see [Logs](/observability/logs). For dashboard graphs and their interpretation, see [Metrics](/observability/metrics). To collect metrics continuously in an external monitoring system, use [metrics export](/observability/export-metrics/overview). For CLI examples, [install and authenticate the Baseten CLI](/reference/cli/baseten/overview). For API examples, set `BASETEN_API_KEY` to a [Baseten API key](/organization/api-keys). Replace the model and deployment ID placeholders with your IDs. ## Fetch and stream logs Pull logs to debug an incident, filter for an error, or use the output in a script. Fetch a time window of up to 7 days: > [tab] UI > > To view logs: > > 1. Sign in to your workspace at > [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the > sidebar. > 2. Select your model. > 3. Choose **Logs**. > 4. Open the dropdown showing the current environment or deployment, then choose your deployment under **Switch deployments**. > > Filter by level or replica, search, or follow live output with the > controls at the top of the logs view. > [tab] Baseten CLI > > Fetch logs from a deployment: > > ```bash Command theme={"system"} > baseten model deployment logs --model-id --deployment-id --since 1h > ``` > > ```txt Output theme={"system"} > [2026-07-07 10:33:58]: Deploy was a success. > [2026-07-07 10:33:44]: (nvsmp) Completed model.load() execution in 6 ms > [2026-07-07 10:33:44]: (nvsmp) Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit) > ``` > > The parenthesized prefix on each line is the replica that emitted it. > > To stream live logs, pass `--tail` instead of a time window: > > ```bash Command theme={"system"} > baseten model deployment logs --model-id --deployment-id --tail > ``` > > ```txt Output theme={"system"} > [2026-07-07 10:33:44]: (nvsmp) Completed model.load() execution in 6 ms > [2026-07-07 10:33:58]: Deploy was a success. > ... > ``` > > Streaming continues until the deployment leaves a runnable state or you > interrupt with Ctrl-C. For machine-readable output, add `--output jsonl` > to stream one JSON log entry per line: > > ```bash Command theme={"system"} > baseten model deployment logs --model-id --deployment-id --tail --output jsonl > ``` > > ```json Output theme={"system"} > {"level":"INFO","message":"Completed model.load() execution in 6 ms","replica":"nvsmp","timestamp":"1783445624380306972"} > {"level":"INFO","message":"Deploy was a success.","replica":"","timestamp":"1783445638726940011"} > ... > ``` > > Narrow the stream with `--jq`: `--jq '.message'` extracts one field per > line, and `--jq 'select(.level=="ERROR") | .message'` keeps only error > lines. > [tab] Management API > > Fetch logs from a deployment: > > ```bash Request theme={"system"} > curl "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/logs" \ > -H "Authorization: Bearer $BASETEN_API_KEY" > ``` > > ```json Response theme={"system"} > { > "logs": [ > { > "timestamp": "1783445638726940011", > "message": "Deploy was a success.", > "replica": "", > "request_id": null, > "level": "INFO" > }, > { > "timestamp": "1783445624380306972", > "message": "Completed model.load() execution in 6 ms", > "replica": "nvsmp", > "request_id": null, > "level": "INFO" > } > ] > } > ``` > > The window defaults to the last 30 minutes; pass `start_epoch_millis` > to widen it, up to 7 days. Timestamps are epoch nanoseconds. The API > fetches fixed windows only; to follow logs live, use the CLI's `--tail`. For more information about filters like log level, search patterns, and request ID, see [`logs`](/reference/cli/baseten/model#logs) and the [logs endpoint](/reference/management-api/deployments/get-deployment-logs). ## Fetch metrics View or fetch a deployment's replica count, request volume, and end-to-end latency quantiles to check its health. > [tab] UI > > To view metrics: > > 1. Sign in to your workspace at > [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the > sidebar. > 2. Select your model. > 3. Choose **Metrics**. > 4. Open the dropdown showing the current environment or deployment, then choose your deployment under **Switch deployments**. > > The metrics view charts request rates by status code, latency > percentiles, and replica counts. Set the window with the time-range > selector at the top. > > To compare individual replicas, follow > [Compare replicas](/observability/metrics#compare-replicas). > [tab] Baseten CLI > > Fetch a deployment's current metrics: > > ```bash Command theme={"system"} > baseten model deployment metrics --model-id --deployment-id > ``` > > ```txt Output theme={"system"} > METRIC QUANTILE STAT VALUE > baseten_replicas_active 1 > baseten_end_to_end_response_time_seconds 0.5 - > baseten_end_to_end_response_time_seconds 0.9 - > baseten_end_to_end_response_time_seconds 0.95 - > baseten_end_to_end_response_time_seconds 0.99 - > baseten_end_to_end_response_time_seconds avg - > ``` > > Latency values show `-` until the deployment has served requests in the > window; once it has, a STATUS column also breaks out request counts per > response code. The default is a current snapshot; pass `--mode summary` > or `--mode series` with `--since` to aggregate over a window instead. > [tab] Management API > > Fetch a deployment's current metrics: > > ```bash Request theme={"system"} > curl "https://api.baseten.co/v1/models/{model_id}/deployments/{deployment_id}/metrics" \ > -H "Authorization: Bearer $BASETEN_API_KEY" > ``` > > ```json Response theme={"system"} > { > "start_epoch_millis": 1783445679297, > "end_epoch_millis": 1783445679297, > "mode": "CURRENT", > "step_seconds": null, > "metric_descriptors": [ > { > "name": "baseten_replicas_active", > "unit_hint": "COUNT", > "kind": "GAUGE", > "label_sets": [{}] > } > ], > "metric_values": [ > { > "start_epoch_millis": 1783445679297, > "values": [[1.0]] > } > ] > } > ``` > > Each `metric_values` entry represents a time window. Within that entry, > `values[i]` corresponds to `metric_descriptors[i]`. The values in each > inner array follow that descriptor's `label_sets` order. For more information about modes, windows, and metric selection, see [`metrics`](/reference/cli/baseten/model#metrics) and the beta [metrics endpoint](/reference/management-api/deployments/get-deployment-metrics). For what each metric means, see [Metrics](/observability/metrics). ## Fetch environment logs and metrics Use [environment](/deployment/environments) scope to follow logs and metrics across promotions. It includes the deployments serving that environment during the selected time window, so you can investigate a rollout without querying each deployment separately. > [tab] UI > > To view an environment's logs and metrics: > > 1. Sign in to your workspace at > [app.baseten.co](https://app.baseten.co) and choose **Dedicated Inference** in the > sidebar. > 2. Select your model. > 3. Choose **Logs** or **Metrics**. > 4. Open the dropdown showing the current environment or deployment, then choose your environment under **Switch environments**. > > The view includes deployments that served that environment during the selected time window. > [tab] Baseten CLI > > To fetch logs, target the environment by name instead of a deployment ID: > > ```bash Command theme={"system"} > baseten model environment logs --model-id --environment production --since 1h > ``` > > ```txt Output theme={"system"} > [2026-07-09 08:49:09]: (w44tt) Application shutdown complete. > [2026-07-09 08:48:41]: Scaling down replicas due to inactivity > [2026-07-09 08:34:09]: (67wjm) Terminated > ... > ``` > > The same windows and filters as deployment logs apply, including > `--tail` for live streaming and `--output jsonl` for machine-readable > output. > > Fetch the environment's current metrics: > > ```bash Command theme={"system"} > baseten model environment metrics --model-id --environment production > ``` > > ```txt Output theme={"system"} > METRIC QUANTILE STAT VALUE > baseten_replicas_active 1 > baseten_end_to_end_response_time_seconds 0.5 - > baseten_end_to_end_response_time_seconds 0.9 - > baseten_end_to_end_response_time_seconds 0.95 - > baseten_end_to_end_response_time_seconds 0.99 - > baseten_end_to_end_response_time_seconds avg - > ``` > > The default is a current snapshot; pass `--mode summary` or > `--mode series` with `--since` to aggregate over a window. In series > mode the window splits at each promotion, so every point reflects the > deployment serving the environment at that time. > [tab] Management API > > Fetch logs from the environment: > > ```bash Request theme={"system"} > curl "https://api.baseten.co/v1/models/{model_id}/environments/production/logs" \ > -H "Authorization: Bearer $BASETEN_API_KEY" > ``` > > ```json Response theme={"system"} > { > "logs": [ > { > "timestamp": "1783612149447676900", > "message": "Application shutdown complete.", > "replica": "w44tt", > "request_id": null, > "level": "INFO" > } > ] > } > ``` > > Fetch the environment's current metrics: > > ```bash Request theme={"system"} > curl "https://api.baseten.co/v1/models/{model_id}/environments/production/metrics" \ > -H "Authorization: Bearer $BASETEN_API_KEY" > ``` > > ```json Response theme={"system"} > { > "start_epoch_millis": 1783968056212, > "end_epoch_millis": 1783968056212, > "mode": "CURRENT", > "step_seconds": null, > "metric_descriptors": [ > { > "name": "baseten_replicas_active", > "unit_hint": "COUNT", > "kind": "GAUGE", > "label_sets": [{}] > } > ], > "metric_values": [ > { > "start_epoch_millis": 1783968056212, > "values": [[1.0]] > } > ] > } > ``` > > The logs endpoint accepts the same time windows and filters as deployment logs. The metrics endpoint accepts the same windows, modes, and metric selection as deployment metrics. For more information, see [`environment logs`](/reference/cli/baseten/model#logs-1) and [`environment metrics`](/reference/cli/baseten/model#metrics-1), and the [environment logs endpoint](/reference/management-api/environments/get-environment-logs) and the beta [environment metrics endpoint](/reference/management-api/environments/get-environment-metrics). ## Next steps Check logs and metrics before and after a deployment change. * [Terminate a stuck replica](/troubleshooting/deployments#issue-a-single-replica-is-stuck-or-unhealthy) you found misbehaving in the logs. * [Scale a deployment](/deployment/manage/scaling) if metrics show sustained queueing or idle replicas. * Interpret dashboard graphs with the [Metrics guide](/observability/metrics), or look up exact names, types, and labels in the [exported metrics reference](/observability/export-metrics/supported-metrics). # Manage deployments Source: https://docs.baseten.co/deployment/manage/overview Scale, promote, inspect, deactivate, and delete Baseten deployments from the dashboard, the Baseten CLI, or the Management API. Learn to operate deployments on Baseten. After you push a model, you'll scale it ahead of traffic, promote new versions, inspect logs and metrics, and deactivate or delete deployments you no longer need. You can manage your deployments using any of the following methods: * **[Dashboard](https://app.baseten.co)**: the Baseten web interface, for interactive changes and checking state. * **[Baseten CLI](/reference/cli/baseten/overview)**: terminal commands for scripting and automation. All commands except the `baseten truss` passthrough support `--output json` and `--jq` filtering. * **[Management API](/reference/management-api/overview)**: REST endpoints for managing models, deployments, and environments from your own code. * **[CI/CD](/deployment/ci-cd)**: automated deploys and promotions from GitHub Actions. ## Find your task * [Scale a deployment](/deployment/manage/scaling): change how much compute a serving deployment uses. Pre-scale before a traffic spike, scale to zero, or wake a scaled-to-zero deployment. * [Manage the deployment lifecycle](/deployment/manage/lifecycle): change whether and where a deployment serves. Promote, deactivate, activate, and delete deployments. * [Label deployments](/deployment/manage/labels): attach deployment metadata for ownership, source control, or CI automation. * [Terminate a stuck replica](/troubleshooting/deployments#issue-a-single-replica-is-stuck-or-unhealthy): remove one unhealthy replica and let the autoscaler replace it. * [Pull logs and metrics](/deployment/manage/logs-and-metrics): inspect a deployment or environment from the dashboard, Baseten CLI, or Management API. * [Download or inspect a deployment's config](#download-or-inspect-a-deployments-config): fetch a deployment's Truss config or source from the Baseten CLI. * [Rename a model](/reference/management-api/models/updates-a-model-by-id): update its name without changing its ID or endpoints. Update `model_name` in your Truss config before your next push. ## Before you start ### Authenticate Sign in with your Baseten account and create an [API key with full access](/organization/api-keys#api-key-types). Inference-only keys can call endpoints but can't manage deployments: > [tab] UI > > **To sign in**: > > 1. Go to [app.baseten.co](https://app.baseten.co). > 2. Enter your email and choose **Continue with email**, or choose > **Continue with Google** or **Continue with GitHub**. > > **To create an API key**: > > 1. In your [workspace](https://app.baseten.co), open **API keys** in > your settings. > 2. Choose **Create API key**. > [tab] Baseten CLI > > **To sign in**: > > ```bash Command theme={"system"} > baseten auth login > ``` > > ```txt Output theme={"system"} > ? How would you like to authenticate? > > Login with Baseten credentials (browser) > Paste an API key > > Browser opened to authenticate... > > If it didn't open, visit: > https://login.baseten.co/device?user_code=ABCD-1234 > > Verification code: ABCD-1234 > ``` > > Complete the sign-in in the browser; the CLI stores a profile that > authenticates later commands. > > For more information, see > [`baseten auth login`](/reference/cli/baseten/auth#login). > > **To create an API key**: > > Create a personal key, tied to your account and its permissions, for > code that calls your deployments. CLI commands authenticate with your > sign-in: > > ```bash Command theme={"system"} > baseten org api-key create --type personal --name