How Baseten works

You can deploy an existing model or train a new one and deploy its checkpoint. This page explains the systems behind both paths: build pipelines, request routing, autoscaling, cold starts, and environments. For product options and starting points, see Baseten overview.

​

Multi-cloud Capacity Management (MCM)

Multi-cloud Capacity Management (MCM) provisions and manages GPUs across cloud providers and geographic regions. When you request hardware, such as an H100 in US-East-1 or a cluster of B200s in a private region, MCM provisions it, configures networking, and monitors its health. Baseten provides a consistent training and inference runtime across the underlying infrastructure. MCM also supports high availability. Deployments run active-active across clusters and clouds. If a region or provider loses capacity, MCM reroutes and reprovisions workloads.

​

Deploy an existing model

To deploy a model, package it with Truss, Baseten’s open-source model packaging tool. Describe the model in a config.yaml (for supported architectures) or a small Python Model class (for custom code), then run truss push to ship it. truss push returns once the upload finishes. For engine-based deployments, compilation can take several minutes. Watch progress in the deployment logs, or wait for the dashboard to show “Active.” For custom model code, Baseten installs your Python dependencies, packages the Model class into a container, and deploys it. Custom builds do not apply Baseten engine optimizations automatically. Each push produces a container image identified by a content hash and stored in Baseten’s container registry. The image is immutable, and an unchanged project reuses the cached image instead of triggering a new build.

​

Train a model

Use Loops to write a Python training loop that calls a dedicated trainer and sampler. Use Training Jobs to run your own training container to completion. To run a training job, define the job in a Python configuration file (typically config.py) using the truss_train SDK, then submit it with truss train push. Baseten provisions GPUs through MCM, runs your training container, and syncs checkpoints to storage as the job progresses. For more information, see the Training Jobs overview.

​

Request routing

Each model gets a dedicated subdomain: https://model-{model_id}.api.baseten.co/. The URL path selects the deployment that handles a request. /production/predict targets the production environment, and /development/predict targets the development deployment. You can also target a deployment by ID or an environment by name. Baseten resolves the target from the URL and routes the request to an active replica. If the deployment has scaled to zero, Baseten starts a replica and holds the request until the model loads. The request uses the same endpoint whether the replica is warm or cold-started. Engine-based deployments expose an OpenAI-compatible API at /v1/chat/completions. Use the OpenAI SDK with the deployment’s base URL and a Baseten API key. Custom model deployments use the predict API, which accepts and returns arbitrary JSON. For long-running workloads, async requests return a request ID immediately. An async request service queues the request. A background worker then calls your model and delivers the result through a webhook. Sync requests get priority when capacity is tight, so background work doesn’t starve real-time traffic.

​

Autoscaling

Baseten’s autoscaler matches replica count to in-flight request load, keeping each replica below its concurrency target. When average load over the autoscaling window (60 seconds by default) crosses the target utilization (70% by default), the autoscaler adds replicas up to the configured maximum. When load drops, the autoscaler waits for scale_down_delay (900 seconds by default), then removes excess replicas at a pace capped by max_scale_down_rate (half of running replicas by default). The timer resets after each reduction, and the cycle repeats until the deployment reaches its target size. This staged reduction prevents scaling changes in response to brief traffic dips. Set min_replica to 0 for scale-to-zero: the deployment incurs no GPU cost when idle, but the next request triggers a cold start. Set min_replica to 1 or higher to keep warm capacity ready, trading cost for lower latency.

​

Cold starts and the Baseten Delivery Network

Loading model weights can dominate cold-start time, especially when weights reach hundreds of gigabytes. The Baseten Delivery Network (BDN) caches model weights across storage, clusters, and nodes. On the first deployment, BDN mirrors model weights from the source repository to Baseten storage. Later cold starts do not depend on the original Hugging Face, S3, or GCS source. When a replica starts, the BDN agent fetches a weight manifest, downloads files through a cache shared by the cluster, and stores them in a node-level cache. BDN deduplicates identical files across models, so a fine-tune that shares files with its base model downloads only the difference. Later cold starts on the same node or cluster can reuse cached weights. Container image streaming also lets the model begin loading weights before the image download completes. BDN serves training jobs the same way. Mount weights and training data into your training container from any supported source, and BDN caches them so subsequent jobs start faster.

​

Environments and promotion

Use a development deployment with scale-to-zero and live reload for fast iteration. When the model is ready for stable traffic, promote an immutable deployment to a named environment, such as production, staging, or canary. Each environment has its own stable URL, autoscaling settings, and metrics. Promoting a new deployment swaps it in for the previous one and inherits the environment’s autoscaling settings. The endpoint URL stays constant when you promote, so your application code doesn’t need to change. Baseten demotes the previous deployment and scales it to zero, so you can roll back by re-promoting it. Promotion reuses the image the deployment was already built with, so it never rebuilds or re-pulls your base image. Rollback works the same way: re-promoting a previous deployment reuses its existing image. To skip the development stage, push directly to an environment with truss push --environment staging. Only one promotion can be active per environment at a time, which prevents conflicting updates. See Deployment concepts for the full set of resource and CI/CD options. These pieces work the same whether you deploy an existing model or train a new one, so the path from prototype to production stays consistent.

​

Next steps