Changelog

Model library

Truss examples

Baseten

API reference

Truss reference 📦

Chains reference ⛓

Workspace

Status

Support

Get all model environments

Timeframe of traffic considered for autoscaling decisions

Autoscaling Window

Number of requests per replica before scaling up

Concurrency Target

Max Replica

Min Replica

Waiting period before scaling down any active replica

Scale Down Delay

AutoscalingSettingsV1

DeploymentStatusV1

Active Replica Count

Autoscaling settings for the deployment. If null, the model has not finished deploying

Time the deployment was created in ISO 8601 format

Created At

The environment associated with the deployment

Environment

Whether the deployment is the development deployment of the model

Is Development

Whether the deployment is the production deployment of the model

Is Production

Model Id

Name

DeploymentV1

Time the environment was created in ISO 8601 format

EnvironmentV1

Ramp Up Duration Seconds

Whether to ramp up traffic while promoting

Ramp Up While Promoting

PromotionSettingsV1

ApiKeyAuth

EnvironmentsV1

Get all environments

Fast, scalable inference in our cloud or yours

Welcome to Baseten!

Deploy your first model

Truss: Package and deploy AI models on Baseten

Overview

Getting Started

LLM with Streaming

Text-to-image

Deploy a language model, with the model weights cached at build time

Fast Cold Starts with Cached Weights

Load a model that requires authentication with Hugging Face

Private Hugging Face Model

Deploy a model with both Python and system dependencies

Model with system packages

A guide to configuring a base image for your truss

Base Docker images

Load model weights without Hugging Face or S3

Model weights

Deploy a model that makes use of pre-process

Pre/post-processing

Streaming output with an LLM

A guide on configuring your truss to use external packages

External (source) packages

How to run your own docker commands during the build stage

Running custom docker commands

Private Hugging Face model

Enable fast cold starts for a model with private Hugging Face weights

Deploy Llama 2 with Caching

A guide to setting concurrency for your model

Request concurrency

Accelerate cold starts by caching your weights

Caching model weights

A guide to using secrets securely in your ML models

Stoaring secrets in Baseten

Get more control by directly using the request object.

Using request objects / Cancellation

Get more control by directly creating the response object.

Returning response objects and SSEs

A guide to leveraging environments in your models

Access model environments

Deployments and environments

Serve your model on the right instance type

Setting GPU resources

Scale from internal testing to the top of Hacker News

Autoscaling

Fixing common problems during model deployment

Troubleshooting

How to call your model

How to stream model output

Run asynchronous inference on deployed models

Async inference user guide

Secure the asynchronous inference results sent to your webhook

Securing async inference

Enforce an output schema on LLM inference

Structured output (JSON mode)

Use an LLM to select amongst provided tools

Function calling (tool use)

How to parse base64 output

How to do model I/O in binary

How to do model I/O with files

Use your Baseten models with tools like LangChain

Baseten model integrations

Fixing common problems during model inference

Chains: A new DX for deploying multi-component ML workflows

Build your first Chain

Glossary of Chains concepts and terminology

Concepts

User Guides

Build a RAG (retrieval-augmented generation) pipeline with  Chains

RAG Chain

Transcribe hours of audio to text in a few seconds

Audio Transcription Chain

Model performance overview

Deploy optimized model inference servers in minutes

Engine Builder overview

Automatically build and deploy a TensorRT-LLM model serving engine

Build your first LLM engine

Use `model.py` to customize engine behavior

Engine control in Python

Configure your TensorRT-LLM inference engine

Engine Builder configuration

Engineering your Truss and application for faster cold starts

How to get faster cold starts

Handle variable throughput with this autoscaling parameter

Setting concurrency

Specs and recommendations for every instance type on Baseten

Instance type reference

Understand the load and performance of your model

Reading model metrics

Export metrics from Baseten to your observability stack

Metrics export overview

Export metrics from Baseten to Prometheus

Export metrics to Prometheus

Export metrics to Datadog

Export metrics from Baseten to Grafana Cloud

Export metrics to Grafana Cloud

Export metrics to New Relic

Metrics support matrix

Monitoring model health

Secure model inference

Share your Baseten workspace with your team

Workspace access control

Best practices for API keys

Securely store and access passwords, tokens, keys, and more

Best practices for secrets

Investigate the prediction flow in detail

Tracing

Manage payments and track overall Baseten usage

Billing and usage

Details on model inference and management APIs

🆕 Inference by environment

Production deployment

Development deployment

Published deployment

ChatCompletions

ChatCompletions (deprecated)

Model endpoint migration guide

Call primary version

Call model version

Wake primary version

Wake model version

Use this endpoint to call the model associated with the specified environment asynchronously.

🆕 Async inference by environment

Use this endpoint to call the [production deployment](/deploy/lifecycle) of your model asynchronously.

Use this endpoint to call the [development deployment](/deploy/lifecycle) of your model asynchronously.

Use this endpoint to call any [published deployment](/deploy/lifecycle) of your model.

Use this endpoint to get the status of an async request.

Get async request status

Use this endpoint to cancel a queued async request.

Cancel async request

Use this endpoint to get the status of a production deployment's async queue.

Use this endpoint to get the async queue status for a model associated with the specified environment.

Environment deployment

Use this endpoint to get the status of a development deployment's async queue.

Use this endpoint to get the status of a published deployment's async queue.

Get all secrets

Creates a new secret or updates an existing secret if one with the provided name already exists. The name and creation date of the created or updated secret is returned.

Inference API

Async Inference API

Management API

Chains Inference API

Get all model environments

Authorizations

Path Parameters

Response