Changelog

Model library

Truss examples

Baseten

API reference

Truss reference

Chains reference 🆕

Workspace

Status

Support

Creates an environment for the specified model and returns the environment.

Create an environment

Timeframe of traffic considered for autoscaling decisions

Autoscaling Window

Number of requests per replica before scaling up

Concurrency Target

Max Replica

Min Replica

Waiting period before scaling down any active replica

Scale Down Delay

AutoscalingSettingsV1

DeploymentStatusV1

Active Replica Count

Autoscaling settings for the deployment. If null, the model has not finished deploying

Time the deployment was created in ISO 8601 format

Created At

The environment associated with the deployment

Environment

Whether the deployment is the development deployment of the model

Is Development

Whether the deployment is the production deployment of the model

Is Production

Model Id

Name

DeploymentV1

Ramp Up Duration Seconds

Whether to ramp up traffic while promoting

Ramp Up While Promoting

PromotionSettingsV1

A request to update autoscaling settings for a deployment. All fields are optional, and we only update ones passed in.

UpdateAutoscalingSettingsV1

UpdatePromotionSettingsV1

CreateEnvironmentRequestV1

ApiKeyAuth

EnvironmentV1

Fast, scalable inference in our cloud or yours

Welcome to Baseten!

Deploy your first model

Model deployment overview

Getting Started

LLM with Streaming

Text-to-image

Deploy a language model, with the model weights cached at build time

Fast Cold Starts with Cached Weights

Load a model that requires authentication with Hugging Face

Private Hugging Face Model

Deploy a model with both Python and system dependencies

Model with system packages

A guide to configuring a base image for your truss

How to use base images

Load model weights without Hugging Face or S3

How to load model weights

Deploy a model that makes use of pre-process

Pre/post-process methods

LLM with streaming output

A guide on configuring your truss to use external packages

How to use external packages

How to run your own docker commands during the build stage

How to run custom docker commands

Private Hugging Face model

Enable fast cold starts for a model with private Hugging Face weights

Deploy Llama 2 with Caching

A guide to setting concurrency for your model

How to configure concurrency

Accelerate cold starts by caching your weights

How to cache model weights

A guide to using secrets securely in your ML models

How to use secrets

Get more control by directly using the request object.

Using request objects / Cancellation

Get more control by directly creating the response object.

Returning response objects and SSEs

Development vs production

Serve your model on the right instance type

How to set GPU resources

Scale from internal testing to the top of Hacker News

How to configure autoscaling

Fixing common problems during model deployment

Deployment troubleshooting

Deploy your model to an environment to enable mature release cycles

🆕 Environments

How to call your model

How to stream model output

Run asynchronous inference on deployed models

Async inference user guide

Secure the asynchronous inference results sent to your webhook

Securing async inference

Enforce an output schema on LLM inference

Structured output (JSON mode)

Use an LLM to select amongst provided tools

Function calling (tool use)

How to parse base64 output

How to do model I/O in binary

How to do model I/O with files

Use your Baseten models with tools like LangChain

Baseten model integrations

Fixing common problems during model invocation

Invocation troubleshooting

A new DX for deploying multi-component ML workflows

Chains overview

Build your first Chain

User Guide

Glossary of Chains concepts and terminology

Concepts

Build a RAG (retrieval-augmented generation) pipeline with  Chains

RAG Chain

Transcribe hours of audio to text in a few seconds

Audio Transcription Chain

Model performance overview

Deploy optimized model inference servers in minutes

Engine Builder overview

Automatically build and deploy a TensorRT-LLM model serving engine

Build your first LLM engine

Use `model.py` to customize engine behavior

Engine control in Python

Configure your TensorRT-LLM inference engine

Engine Builder configuration

Engineering your Truss and application for faster cold starts

How to get faster cold starts

Handle variable throughput with this autoscaling parameter

How to set concurrency

Specs and recommendations for every instance type on Baseten

Instance type reference

Understand the load and performance of your model

Reading model metrics

Export metrics from Baseten to your observability stack

Metrics export overview

Export metrics from Baseten to Prometheus

Export metrics to Prometheus

Export metrics to Datadog

Export metrics from Baseten to Grafana Cloud

Export metrics to Grafana Cloud

Export metrics to New Relic

Metrics support matrix

Monitoring model health

Secure model inference

Share your Baseten workspace with your team

Workspace access control

Best practices for API keys

Securely store and access passwords, tokens, keys, and more

Best practices for secrets

Manage payments and track overall Baseten usage

Billing and usage

Details on model inference and management APIs

Production deployment

Use this endpoint to call the model associated with the specified environment.

🆕 Inference by environment

Development deployment

Published deployment

ChatCompletions

ChatCompletions (deprecated)

Model endpoint migration guide

Call primary version

Call model version

Wake primary version

Wake model version

Use this endpoint to call the [production deployment](/deploy/lifecycle) of your model asynchronously.

Use this endpoint to call the model associated with the specified environment asynchronously.

🆕 Async inference by environment

Use this endpoint to call the [development deployment](/deploy/lifecycle) of your model asynchronously.

Use this endpoint to call any [published deployment](/deploy/lifecycle) of your model.

Use this endpoint to get the status of an async request.

Get async request status

Use this endpoint to cancel a queued async request.

Cancel async request

Get all secrets

Creates a new secret or updates an existing secret if one with the provided name already exists. The name and creation date of the created or updated secret is returned.

Upsert a secret

Get all models

Get a model by ID

Get all model deployments

Get all environments

Gets an environment's details and returns the environment.

Get environment

Updates an environment's settings and returns the updated environment.

Update environment

Gets a model's production deployment and returns the deployment.

Gets a model's development deployment and returns the deployment.

Gets a model's deployment by id and returns the deployment.

Any deployment by ID

Updates a production deployment's autoscaling settings and returns the update status.

Updates a development deployment's autoscaling settings and returns the update status.

Updates a deployment's autoscaling settings and returns the update status.

Promotes an existing deployment to an environment and returns the promoted deployment.

🆕 Promote to environment

Creates a new production deployment from the development deployment, the currently building deployment is returned.

Promotes an existing deployment to production and returns the same deployment.

Activates an inactive deployment associated with an environment and returns the activation status.

🆕 Activate environment deployment

Activates an inactive production deployment and returns the activation status.

Inference API

Async Inference API

Management API

Chains Inference API

Create an environment

Authorizations

Path Parameters

Body

Response