Baseten overview

Baseten is a platform for model inference and training. You can call hosted models through an OpenAI-compatible API, deploy your own models on dedicated infrastructure, or train models and serve the resulting checkpoints. Baseten manages model containers, GPU capacity across clouds and regions, scaling, and observability. Its inference engines optimize supported model architectures for production traffic.

​

Choose how to start

To point a coding agent at Model APIs, see Coding agents. To let an agent manage your workspace, see Connect your coding agent. For a dedicated speech endpoint, transcribe speech in batch or real time with a model from the Model Library.

​

Model APIs

Model APIs expose hosted endpoints for a curated set of models. They support the OpenAI Chat Completions API and the Anthropic Messages API in beta, so you can use familiar client SDKs. Choose Model APIs when a supported model fits your application and you want to start with an API call. Baseten manages the model, engine, hardware, and scaling.

​

Dedicated inference

Deploy an open-source model, private checkpoint, or custom model on dedicated infrastructure. Define the model with a configuration file, a Python model class, or a custom Docker server, then deploy it with the Baseten CLI. See Develop a model on Baseten for each approach. For supported architectures, a config.yaml can define the model, hardware, and inference engine without custom serving code. Use a Python model class or a custom Docker server when you need custom preprocessing, postprocessing, dependencies, or server behavior. Dedicated deployments give you control over the model, hardware, autoscaling settings, and release lifecycle. They also support stable environments for development, staging, and production.

​

Inference engines

Baseten maintains inference engines for common model architectures:

• Engine-Builder-LLM compiles dense text-generation models with TensorRT-LLM.
• BIS-LLM serves large mixture-of-experts models with distributed inference and KV-aware routing.
• BEI serves embedding, reranking, and classification models.

You can select an engine in config.yaml, let Baseten select one for a supported architecture, or run a server such as vLLM or SGLang in a custom Docker container.

​

Training

Baseten has two ways to train models:

• Loops provides dedicated trainers and samplers for supervised fine-tuning and reinforcement learning on supported base models.
• Training Jobs runs your framework and training code in your own container on dedicated GPU clusters.

Both paths store checkpoints that you can deploy to the Baseten inference stack. See Training on Baseten to compare them.

​

Sandboxes

Sandboxes give agents an isolated environment to run code and work with files. Each sandbox starts from an image. Set a lifetime so Baseten deletes the sandbox when it expires, and save the task’s results outside the sandbox before then.

​

Production operations

Every dedicated deployment includes logs, metrics, request traces, autoscaling, and environment promotion. You can export observability data to Datadog, Prometheus, Grafana, or New Relic. Store secrets in the workspace instead of in source control. Multi-cloud Capacity Management provisions GPUs across cloud providers and regions. Hosting options support private networking, workload isolation, infrastructure ownership, and specific data regions. Baseten is SOC 2 Type II and HIPAA compliant. Use regional environments to constrain replica placement and inference routing for architectures with data-residency requirements, or regional deployments to select a region for a single deployment.

​

Baseten for Model Labs

Baseten for Model Labs helps model creators serve and distribute models to customers. Access is enabled per workspace. To get access, talk to us.

​

Next steps