> ## Documentation Index
> Fetch the complete documentation index at: https://docs.baseten.co/llms.txt
> Use this file to discover all available pages before exploring further.
# How Baseten works
> The moving parts behind training, deployment, request routing, autoscaling, and environment promotion on Baseten.
You can deploy an existing model or train a new one and deploy its checkpoint. This page explains the systems behind both paths: build pipelines, request routing, autoscaling, cold starts, and environments. For product options and starting points, see [Baseten overview](/overview).
## Multi-cloud Capacity Management (MCM)
Multi-cloud Capacity Management (MCM) provisions and manages GPUs across cloud providers and geographic regions.
When you request hardware, such as an H100 in US-East-1 or a cluster of B200s in a private region, MCM provisions it, configures networking, and monitors its health. Baseten provides a consistent training and inference runtime across the underlying infrastructure.
MCM also supports high availability. Deployments run active-active across clusters and clouds. If a region or provider loses capacity, MCM reroutes and reprovisions workloads.
## Deploy an existing model
To deploy a model, package it with [Truss](https://pypi.org/project/truss/), Baseten's open-source model packaging tool. Describe the model in a `config.yaml` (for supported architectures) or a small Python `Model` class (for custom code), then run `baseten model push` to ship it.
`baseten model push` validates your `config.yaml`, archives your project directory, and uploads it to cloud storage. Baseten receives the archive and starts the build.
For [Engine-Builder-LLM](/engines/engine-builder-llm/overview), Baseten downloads model weights from the source repository (Hugging Face, S3, or GCS) and compiles them with TensorRT-LLM. Compilation builds optimized CUDA kernels for the target GPU architecture, applies quantization if configured, and sets up tensor parallelism across multiple GPUs.
Baseten packages the compiled engine, runtime configuration, and serving infrastructure into a container, deploys it to GPU infrastructure, and exposes it as an API endpoint.
`baseten model push` returns once the upload finishes. For engine-based deployments, compilation can take several minutes. Watch progress in the deployment logs, or wait for the dashboard to show "Active."
For [custom model code](/development/model/model-class), Baseten installs your Python dependencies, packages the `Model` class into a container, and deploys it. Custom builds do not apply Baseten engine optimizations automatically.
Each push produces a container image identified by a content hash and stored in Baseten's container registry. The image is immutable, and an unchanged project reuses the cached image instead of triggering a new build.
## Train a model
Use [Loops](/loops/overview) to write a Python training loop that calls a dedicated trainer and sampler. Use [Training Jobs](/training/overview) to run your own training container to completion.
To run a training job, define the job in a Python configuration file (typically `config.py`) using the [`truss_train` SDK](/reference/sdk/training), then submit it with `baseten train push --config config.py`. Baseten provisions GPUs through MCM, runs your training container, and syncs checkpoints to storage as the job progresses.
`baseten train push --config config.py` packages your training config, uploads it to Baseten, and starts the job on the hardware you specified (H100 or H200, single-node or multi-node). Your training code can use Axolotl, TRL, VeRL, Megatron, or any other framework you bundle into the container.
Baseten runs your training container on the provisioned GPUs. As your training code writes checkpoints to the configured directory, Baseten uploads them to durable storage. If the job fails or you stop it, you can still use the most recent checkpoint.
`baseten train checkpoint deploy --job-id ` constructs a Truss `config.yaml` from the checkpoint, packages it as a deployment, and exposes an API endpoint. From there, the deployment behaves like any other model on Baseten.
For more information, see the [Training Jobs overview](/training/overview).
## Request routing
Each model gets a dedicated subdomain: `https://model-{model_id}.api.baseten.co/`. The URL path selects the deployment that handles a request. `/production/predict` targets the production environment, and `/development/predict` targets the development deployment. You can also target a deployment by ID or an environment by name.
Baseten resolves the target from the URL and routes the request to an active replica. If the deployment has scaled to zero, Baseten starts a replica and holds the request until the model loads. The request uses the same endpoint whether the replica is warm or cold-started.
Engine-based deployments expose an [OpenAI-compatible API](/reference/inference-api/chat-completions) at `/v1/chat/completions`. Use the OpenAI SDK with the deployment's base URL and a Baseten API key. Custom model deployments use the [predict API](/reference/inference-api/overview), which accepts and returns arbitrary JSON.
For long-running workloads, [async requests](/inference/async) return a request ID immediately. An async request service queues the request. A background worker then calls your model and delivers the result through a webhook. Sync requests get priority when capacity is tight, so background work doesn't starve real-time traffic.
## Autoscaling
Baseten's autoscaler matches replica count to in-flight request load, keeping each replica below its [concurrency target](/deployment/autoscaling/overview).
When average load over the autoscaling window (60 seconds by default) crosses the target utilization (70% by default), the autoscaler adds replicas up to the configured maximum.
When load drops, the autoscaler waits for `scale_down_delay` (900 seconds by default), then removes excess replicas at a pace capped by `max_scale_down_rate` (half of running replicas by default). The timer resets after each reduction, and the cycle repeats until the deployment reaches its target size. This staged reduction prevents scaling changes in response to brief traffic dips.
Set [`min_replica`](/deployment/autoscaling/overview) to 0 for scale-to-zero: the deployment incurs no GPU cost when idle, but the next request triggers a cold start. Set `min_replica` to 1 or higher to keep warm capacity ready, trading cost for lower latency.
## Cold starts and the Baseten Delivery Network
Loading model weights can dominate cold-start time, especially when weights reach hundreds of gigabytes. The [Baseten Delivery Network (BDN)](/development/model/bdn) caches model weights across storage, clusters, and nodes.
On the first deployment, BDN mirrors model weights from the source repository to Baseten storage. Later cold starts do not depend on the original Hugging Face, S3, or GCS source. When a replica starts, the BDN agent fetches a weight manifest, downloads files through a cache shared by the cluster, and stores them in a node-level cache. BDN deduplicates identical files across models, so a fine-tune that shares files with its base model downloads only the difference.
Later cold starts on the same node or cluster can reuse cached weights. Container image streaming also lets the model begin loading weights before the image download completes.
BDN serves training jobs the same way. Mount weights and training data into your training container from any supported source, and BDN caches them so subsequent jobs start faster.
## Environments and promotion
Use a development deployment with scale-to-zero and live reload for fast iteration. When the model is ready for stable traffic, promote the development deployment directly to a named [environment](/deployment/environments), such as production or staging. Promotion creates a published deployment from the current development state.
Each environment has its own stable URL, autoscaling settings, and metrics. When you promote a new deployment, Baseten applies the environment's autoscaling settings and routes its endpoint to the new deployment according to its promotion settings. The endpoint URL stays constant, so your calling code doesn't need to change. After the promotion, Baseten handles the previous deployment according to the environment's cleanup strategy.
Promoting a published deployment reuses its existing image. Re-promoting a previous published deployment for rollback also reuses that image. Promoting a development deployment creates a new published deployment from its current state and triggers an image build.
To skip the development stage, push directly to an environment with `baseten model push --environment staging`. Only one promotion can be active per environment at a time, which prevents conflicting updates. See [Deployment concepts](/deployment/concepts) for the full set of resource and CI/CD options.
These pieces work the same whether you deploy an existing model or train a new one, so the path from prototype to production stays consistent.
## Next steps
Deploy a Hugging Face model with `config.yaml` and the Baseten CLI.
Run a fine-tune or pre-train and deploy the checkpoint to an endpoint.