# Baseten: Documentation ## Documentation ### Get started - [Baseten overview](https://docs.baseten.co/overview.md): Run hosted models, deploy custom models, and train models on production GPU infrastructure. - [Quickstart](https://docs.baseten.co/quickstart.md): Call a shared Model API endpoint without deploying or managing infrastructure. - [How Baseten works](https://docs.baseten.co/concepts/howbasetenworks.md): The moving parts behind training, deployment, request routing, autoscaling, and environment promotion on Baseten. - [Manage Baseten with an agent](https://docs.baseten.co/agent-setup.md): Install the Baseten skill and MCP servers so your agent can manage your Baseten workspace and search these docs. ### Model APIs - [Model APIs](https://docs.baseten.co/inference/model-apis/overview.md): Call hosted language and decision models, compare capabilities, and connect applications or coding agents. - [Decision models](https://docs.baseten.co/inference/model-apis/decision-models.md): Classify inputs, evaluate images, choose a route, and score requests with decision models on Model APIs - [Reasoning](https://docs.baseten.co/inference/model-apis/reasoning.md): Control reasoning for reasoning-capable models - [Vision](https://docs.baseten.co/inference/model-apis/vision.md): Send images and videos alongside text to vision-capable models - [Audio](https://docs.baseten.co/inference/model-apis/audio.md): Send audio recordings alongside text prompts to audio-capable models on Model APIs - [Web search](https://docs.baseten.co/inference/model-apis/web-search.md): Get grounded answers in Model APIs with server-side tool execution - [Use models in coding agents](https://docs.baseten.co/inference/model-apis/coding-agents.md): Call Model APIs from Pi, Droid, or OpenCode. Connect Claude Code, Codex CLI, or Pi with Baseten Switch. - [Pricing and limits](https://docs.baseten.co/inference/model-apis/pricing-and-limits.md): Understand Model API pricing and rate limits, then track usage and costs - [Deprecation](https://docs.baseten.co/inference/model-apis/deprecation.md): Baseten's deprecation policy for Model APIs ### Baseten Grid - [Baseten Grid](https://docs.baseten.co/grid/overview.md): Connect your organization's coding agents to models on Baseten and frontier providers, with model access by team, monthly spend limits, and usage tracking. - [Grid supported coding agents and models](https://docs.baseten.co/grid/supported-agents-and-models.md): The coding agents, platforms, model providers, and model capabilities that Baseten Grid supports. #### For admins - [Set up Grid for your organization](https://docs.baseten.co/grid/set-up.md): Prepare Grid for your organization: organize teams, connect model providers, choose each team's models, and set spend limits. - [Set spend limits and view usage](https://docs.baseten.co/grid/manage.md): Set per-member and team-wide spend limits, change a developer's monthly limit, and see spend by developer, model, and provider. - [Roll out Grid to developers](https://docs.baseten.co/grid/roll-out.md): Send developers a ready-to-edit announcement, script setup across machines, and know what setup changes in each coding agent. #### For developers - [Connect your coding agent to Grid](https://docs.baseten.co/grid/developers/set-up.md): Run baseten grid setup to point Claude Code, Codex, OpenCode, or Pi at your team's Grid models. - [Check your Grid usage](https://docs.baseten.co/grid/developers/usage.md): See your Grid spend for the month against your limit, understand what happens at the limit, and manage your Grid API keys. ### Inference - [Overview](https://docs.baseten.co/inference/overview.md): Inference on Baseten: Model APIs, self-deployed models, how responses are delivered, structured outputs, tool calling, and client configuration. - [Call your model](https://docs.baseten.co/inference/calling-your-model.md): Choose your deployed model's interface, send a request, and read its response. - [Streaming](https://docs.baseten.co/inference/streaming.md): Read incremental model output and check completion for your API protocol. - [Async inference](https://docs.baseten.co/inference/async.md): Run asynchronous inference on deployed models - [Structured outputs](https://docs.baseten.co/inference/structured-outputs.md): Generate schema-constrained JSON and handle incomplete or refused responses. - [JSON mode](https://docs.baseten.co/inference/json-mode.md): Constrain model output to syntactically valid JSON - [Function calling](https://docs.baseten.co/inference/function-calling.md): Define tools, execute model-selected calls, and return results through Chat Completions or Responses. - [Configure HTTP clients](https://docs.baseten.co/inference/http-client-configuration.md): Configure connection pooling, retries, and timeouts for reliable inference requests at scale. - [Integrations](https://docs.baseten.co/inference/integrations.md): Choose a coding agent, application framework, gateway, or voice integration for Baseten. #### Speech and audio - [Speech and audio](https://docs.baseten.co/inference/audio/overview.md): Choose a path for transcription, speech generation, speaker diarization, and audio understanding on Baseten. - [Transcribe speech](https://docs.baseten.co/inference/audio/quickstart.md): Deploy Qwen3-ASR for batch transcription over HTTP or real-time transcription over WebSocket. - [Voice inference performance](https://docs.baseten.co/inference/audio/performance.md): Measure speech latency and quality, choose concurrency targets, and evaluate deployment cost under representative traffic. #### Files and binary data - [Model I/O in binary](https://docs.baseten.co/inference/output-format/binary.md): Decode and save binary model output - [Model I/O with files](https://docs.baseten.co/inference/output-format/files.md): Send JSON, file bytes, or URLs to custom Truss models and save the results. ### Frontier Gateway - [Baseten Frontier Gateway](https://docs.baseten.co/frontier-gateway/overview.md): Offer an inference API to your customers with model routing, API keys, usage limits, and billing events. - [Get started](https://docs.baseten.co/frontier-gateway/get-started.md): Create an endpoint, create a group, mint an API key, and call your model through the gateway. - [Manage endpoints](https://docs.baseten.co/frontier-gateway/endpoints.md): Create and manage the endpoints that route Frontier Gateway traffic to Baseten, third-party, and custom targets. - [Manage groups and API keys](https://docs.baseten.co/frontier-gateway/api-keys.md): Walk the full lifecycle: create groups, build a hierarchy, mint and revoke API keys, and delete groups when a customer churns. - [Rate and usage limits](https://docs.baseten.co/frontier-gateway/rate-limits.md): Per-group, per-model token and request limits, two inheritance modes, and how Frontier Gateway computes the effective limits the runtime enforces. - [Billing events](https://docs.baseten.co/frontier-gateway/billing-events.md): Meter Frontier Gateway usage from per-request billing events: fetch them on demand with the events API, or receive them as signed webhooks. - [Call a model through Frontier Gateway](https://docs.baseten.co/frontier-gateway/calling-your-model.md): Call models through Frontier Gateway with a federated API key. ### Training - [Training on Baseten](https://docs.baseten.co/training/index.md): Post-train supported language models with Loops, run any workload with Training Jobs, or scale the rollouts of your own RL trainer with RL Rollouts. #### Learn post-training - [Post-train a model](https://docs.baseten.co/post-training/overview.md): Go from a task to a deployed post-trained model: decide whether to train, define quality, choose a model and method, run, evaluate, and ship. - [Decide whether to post-train](https://docs.baseten.co/post-training/decide.md): Check whether post-training is the smallest fix for your task: rule out prompt and pipeline fixes, confirm the task and data are ready, and estimate the payoff. - [Evaluate a model](https://docs.baseten.co/post-training/evaluate.md): Measure whether a model or a checkpoint does your task, using rules, a code check, a validated judge, and paired comparisons. - [Choose a base model](https://docs.baseten.co/post-training/choose-a-model.md): Narrow Loops base models by license, sequence length, availability mode, and serving latency and cost, then baseline the candidates on your evaluation. - [Choose SFT or RL](https://docs.baseten.co/post-training/choose-a-method.md): Choose supervised fine-tuning when you have answers to imitate and reinforcement learning when you can score outcomes, and combine them when a task needs both. - [Ship a post-trained model](https://docs.baseten.co/post-training/ship.md): Confirm a run learned, compare the checkpoint with its baseline and the model it replaces, deploy it, roll it out with a way back, and feed production back into the next run. #### Loops - [Loops](https://docs.baseten.co/loops/overview.md): Write Python scripts for LoRA fine-tuning on managed training and sampling infrastructure. - [Loops quickstart](https://docs.baseten.co/loops/quickstart.md): Train Qwen3.6-35B-A3B with RL on serverless Loops, save resumable training state, and deploy the tuned model for inference. - [Concepts](https://docs.baseten.co/loops/concepts.md): How Loops sessions, runs, trainers, samplers, and checkpoints fit together. - [Supported models](https://docs.baseten.co/loops/supported-models.md): Compare Loops base models by availability mode, sequence length, and vision support, then check your workspace's access. ##### Guides - [Choose where to run your Loops script](https://docs.baseten.co/loops/where-to-run.md): Run your Loops training script on your local machine to start, then on Baseten with baseten loops exec for long or unattended runs. - [Prepare data for supervised fine-tuning](https://docs.baseten.co/loops/sft-data.md): Choose trustworthy targets, format conversations, decide which turns train, match serving, and render every row before you pay for a trainer. - [Design RL environments and rewards](https://docs.baseten.co/loops/rl-environments.md): Define tasks, an agent, and a grader for reinforcement learning on Loops, and size tasks so every group has something to learn from. - [Check whether an RL run is learning](https://docs.baseten.co/loops/rl-run-health.md): Read reward, discarded groups, KL divergence between the sampler and trainer, and transcripts to tell a learning RL run from a stalled or exploited one. - [Deactivate a run](https://docs.baseten.co/loops/deactivate-a-run.md): Save the checkpoints you need, then deactivate a Loops run to end it and its paired sampler in either availability mode. - [Deploy a checkpoint](https://docs.baseten.co/loops/deploy-checkpoints.md): Turn a Loops sampler checkpoint into a dedicated inference deployment and call it. - [Migrate from Tinker](https://docs.baseten.co/loops/tinker-compatibility.md): Move an existing Tinker training script to Loops by installing the compatibility package. ##### Features - [Hyperparameters](https://docs.baseten.co/loops/hyperparameters.md): Choose run, optimizer, loss, and sampling settings, and apply a learning-rate schedule in your training loop. - [Loss functions](https://docs.baseten.co/loops/loss-functions.md): Loss functions supported by the Loops trainer, with data shapes and minimal snippets. - [Router replay](https://docs.baseten.co/loops/router-replay.md): Replay the sampler's expert choices in the trainer, so RL on mixture-of-experts models trains each token through the experts that generated it. - [Vision](https://docs.baseten.co/loops/vision.md): Train vision-language models on image and text pairs: which models accept images, how to build an image input, and how to mask its loss targets. ###### Availability modes - [Availability modes](https://docs.baseten.co/loops/availability-modes/overview.md): Compare the serverless and dedicated availability modes, see how Loops chooses one for each run, and choose one yourself. - [Serverless Loops](https://docs.baseten.co/loops/availability-modes/serverless.md): Train and sample without managing deployments, and pay for the tokens you use. - [Dedicated Loops](https://docs.baseten.co/loops/availability-modes/dedicated.md): Run Loops trainers and samplers on GPU deployments allocated to your run, and scale them with replicas, spot capacity, and warm reuse. #### RL Rollouts - [RL Rollouts](https://docs.baseten.co/rl-rollouts/overview.md): Run your own RL trainer and use Baseten to serve rollouts, with weight updates from checkpoints in your bucket. - [Deploy RL Rollouts](https://docs.baseten.co/rl-rollouts/deploy.md): Publish a checkpoint to Amazon S3 or Google Cloud Storage, deploy a rollout model, and load new weights without redeploying. #### Jobs - [Training Jobs](https://docs.baseten.co/training/overview.md): Bring your own container and training code, run it on managed GPUs, and deploy any checkpoint to production inference. - [Training Jobs quickstart](https://docs.baseten.co/training/getting-started.md): Run your first training job and deploy it to production. - [Train on your data with Training Jobs](https://docs.baseten.co/training/your-own-data.md): Swap the tutorial's dataset, config, and hardware for your real fine-tuning workload. - [Lifecycle](https://docs.baseten.co/training/lifecycle.md): Understanding the different states and transitions in a Baseten training job's lifecycle. - [Management](https://docs.baseten.co/training/management.md): How to monitor, manage, and interact with your training projects and jobs. - [Slurm workstations](https://docs.baseten.co/training/slurm.md): Launch a multi-node Slurm cluster on Baseten training infrastructure with a single CLI command. - [Load checkpoints](https://docs.baseten.co/training/loading.md): Resume training from existing checkpoints to continue where you left off. - [Serve your trained model](https://docs.baseten.co/training/deployment.md): How to deploy checkpoints from Training Jobs as usable models. - [Deploy with optimized inference engines](https://docs.baseten.co/training/deploy-with-engine-builder.md): Deploy model checkpoints from Training Jobs directly to an inference engine without downloading or re-uploading weights. ##### Concepts - [Building blocks](https://docs.baseten.co/training/concepts/basics.md): Learn how to get up and running with Training Jobs - [Cache](https://docs.baseten.co/training/concepts/cache.md): Learn how to use the training cache to speed up your training iterations by persisting data between jobs. - [Checkpoints](https://docs.baseten.co/training/concepts/checkpoints.md): Learn how to use Baseten's checkpointing feature to manage model checkpoints and avoid disk errors during training. - [Storage and data ingestion](https://docs.baseten.co/training/concepts/storage.md): Load model weights and training data into Baseten training containers through BDN, S3, Hugging Face, and GCS. - [Multinode training](https://docs.baseten.co/training/concepts/multinode.md): Learn how to configure and run multinode training jobs with Training Jobs. ##### Remote access - [Remote access](https://docs.baseten.co/training/remote-access.md): Connect to running training jobs from your local machine to debug, inspect state, and develop interactively. - [SSH access](https://docs.baseten.co/training/ssh.md): Connect to training containers directly from your terminal with standard SSH. - [VS Code and Cursor remote tunnels](https://docs.baseten.co/training/interactive-sessions.md): Connect to training containers for remote debugging and development through VS Code or Cursor Remote Tunnels. ### Sandboxes - [Sandboxes](https://docs.baseten.co/sandboxes/overview.md): Give each user or project in your agentic application an isolated environment where an agent can build software, run research, and work with files. - [Manage sandbox images](https://docs.baseten.co/sandboxes/manage-images.md): Choose a built-in image or create your own, then list, version, and delete your team's sandbox images. - [Preview a sandbox application](https://docs.baseten.co/sandboxes/previews.md): Open a web application running in a sandbox in a browser, or share it, through a public or private HTTPS URL. - [Sandbox volumes](https://docs.baseten.co/sandboxes/volumes.md): Keep files across sandboxes by creating, mounting, growing, and deleting persistent volumes. - [Create a sandbox access token](https://docs.baseten.co/sandboxes/authentication.md): Authenticate sandbox requests and MCP connections with a short-lived sandbox access token instead of your Baseten API key. #### Manage sandboxes - [Create and manage sandboxes](https://docs.baseten.co/sandboxes/manage.md): Create, inspect, list, update, and delete sandboxes with an SDK, the Baseten CLI, the Management API, or the dashboard. - [Configure sandboxes](https://docs.baseten.co/sandboxes/configure.md): Choose a sandbox's memory and vCPUs, set environment variables and labels, and select the team that owns it. - [Run commands and work with files](https://docs.baseten.co/sandboxes/run-commands.md): Run commands and background processes, read and write files, and call HTTP services in a running sandbox with an SDK, the CLI, or the sandbox runtime API. - [Sandbox lifecycle and expiration](https://docs.baseten.co/sandboxes/lifecycle.md): Sandboxes go on standby when idle and resume on the next request. Track deployment status, set an expiration policy, and keep files before deletion removes them. #### Observability - [Inspect sandbox logs](https://docs.baseten.co/sandboxes/logs.md): Read a command's process output, a sandbox's runtime logs, and image build logs. - [Inspect sandbox metrics](https://docs.baseten.co/sandboxes/metrics.md): Track a sandbox's requests, error rate, CPU, and memory in the dashboard to investigate workload behavior. ### Development - [Develop a model on Baseten](https://docs.baseten.co/development/model/overview.md): Package, configure, and iterate on a model with Truss at whatever level of control your model needs. - [Configure a model server](https://docs.baseten.co/development/model/build-your-first-model.md): Deploy a Hugging Face model with vLLM and your own serving configuration. - [Deploy and iterate](https://docs.baseten.co/development/model/deploy-and-iterate.md): Test Python Truss edits with live patching, then validate and release a published deployment. - [Custom Docker containers](https://docs.baseten.co/development/model/custom-server.md): Deploy custom Docker containers to run inference servers like vLLM, SGLang, Triton, or any containerized application. - [Performance optimization](https://docs.baseten.co/development/model/performance-optimization.md): Optimize model latency, throughput, and cost with Baseten engines #### Configuration - [Configuration](https://docs.baseten.co/development/model/configuration.md): Configure model dependencies, resources, and build environment in config.yaml - [Dependencies](https://docs.baseten.co/development/model/dependencies.md): Declare everything your model needs to build and run: Python packages, build commands, base images, and private registries. #### Model class - [The Model class](https://docs.baseten.co/development/model/model-class.md): Write custom Python in model/model.py to control how your model loads, runs inference, and shapes responses. - [Streaming and endpoints](https://docs.baseten.co/development/model/streaming-and-endpoints.md): Stream model output, expose /v1 HTTP endpoints, and handle raw requests in custom Truss model code. - [Access model environments](https://docs.baseten.co/development/model/environments.md): Configure model behavior based on environment #### Transports - [WebSockets](https://docs.baseten.co/development/model/websockets.md): Enable real-time, streaming, bidirectional communication using WebSockets for Truss models and Chains. - [gRPC](https://docs.baseten.co/development/model/grpc.md): Invoke your model over gRPC. #### Weights and caching - [Baseten Delivery Network](https://docs.baseten.co/development/model/bdn.md): Mirror and cache model weights to reduce deployment and cold-start download time. - [Cached weights](https://docs.baseten.co/development/model/model-cache.md): Accelerate cold starts and availability by prefetching and caching your weights. - [Startup cache](https://docs.baseten.co/development/model/startup-cache.md): Reuse the kernels a deployment's first replica compiles so later replicas start without recompiling - [Runtime caching](https://docs.baseten.co/development/model/runtime-caching.md): Cache files your model writes at runtime so other replicas reuse them #### Volumes - [Volumes](https://docs.baseten.co/development/volumes/overview.md): Publish model weights and data once as immutable BDN volumes, and mount them read-only into any deployment. - [Manage volumes](https://docs.baseten.co/development/volumes/manage.md): Publish volume versions, manage tags and expiration, and roll back deployments. - [Sync volumes from remote sources](https://docs.baseten.co/development/volumes/sync.md): Copy a Hugging Face repository, an object storage prefix, or a Baseten Training checkpoint into a BDN volume as a new version. #### Operations - [Secrets](https://docs.baseten.co/development/model/secrets.md): Use secrets securely in your models - [Egress restrictions](https://docs.baseten.co/development/model/egress-restrictions.md): Restrict a deployment's outbound network traffic to an allow list of IP ranges and domains. - [Health checks](https://docs.baseten.co/development/model/health-checks.md): Customize the health of your deployments. #### Develop a Chain - [Overview](https://docs.baseten.co/development/chain/overview.md): Build and deploy multi-step, multi-model inference pipelines with Chains. - [Concepts](https://docs.baseten.co/development/chain/concepts.md): Glossary of Chains concepts and terminology - [Your first Chain](https://docs.baseten.co/development/chain/getting-started.md): Build and deploy two example Chains - [Architecture and design](https://docs.baseten.co/development/chain/design.md): How to structure your Chainlets - [Local development](https://docs.baseten.co/development/chain/localdev.md): Iterating, Debugging, Testing, Mocking - [Deploy](https://docs.baseten.co/development/chain/deploy.md): Deploy your Chain on Baseten - [Invocation](https://docs.baseten.co/development/chain/invocation.md): Call your deployed Chain - [Watch](https://docs.baseten.co/development/chain/watch.md): Live-patch deployed code - [Subclassing](https://docs.baseten.co/development/chain/subclassing.md): Modularize and re-use Chainlet implementations - [Streaming](https://docs.baseten.co/development/chain/streaming.md): Streaming outputs, reducing latency, SSEs - [Binary I/O](https://docs.baseten.co/development/chain/binaryio.md): Performant serialization of numeric data - [Error handling](https://docs.baseten.co/development/chain/errorhandling.md): Understanding and handling Chains errors - [Truss integration](https://docs.baseten.co/development/chain/stub.md): Integrate deployed Truss models with stubs - [Add an existing Truss model to a Chain](https://docs.baseten.co/development/chain/truss-chainlets.md): Wrap a Truss directory as a chainlet with TrussChainlet, then call it over HTTP or WebSockets with TrussHandle. - [Engine Builder LLM models](https://docs.baseten.co/development/chain/engine-builder-models.md): Engine-Builder LLM models are pre-trained models that are optimized for specific inference tasks. ### Deployment - [Concepts](https://docs.baseten.co/deployment/concepts.md): Deployments, environments, resources, autoscaling, and CI/CD on Baseten. - [Deployments](https://docs.baseten.co/deployment/deployments.md): Understand published and development deployments, how they relate to environments, and which lifecycle action to use. - [Environments](https://docs.baseten.co/deployment/environments.md): Manage your model's release cycles with environments. - [Rolling deployments](https://docs.baseten.co/deployment/rolling-deployments.md): Gradually shift traffic to a new deployment with replica-based rolling deployments. - [Regional environments](https://docs.baseten.co/deployment/regional-environments.md): Constrain replicas and route inference requests within a designated geographic region to meet data-residency requirements. - [Regional deployments](https://docs.baseten.co/deployment/regional-deployments.md): Select a Baseten region for a deployment for compliance, data residency, or lower latency. - [Resources](https://docs.baseten.co/deployment/resources.md): Select and configure CPU, GPU, memory, and multi-node resources for model deployments. - [CI/CD](https://docs.baseten.co/deployment/ci-cd.md): Deploy models from GitHub with the Baseten CLI or the Truss Push Action. #### Manage - [Manage deployments](https://docs.baseten.co/deployment/manage/overview.md): Scale, promote, inspect, deactivate, and delete Baseten deployments from the dashboard, the Baseten CLI, or the Management API. - [Scale a deployment](https://docs.baseten.co/deployment/manage/scaling.md): Update autoscaling settings and scale to zero through the dashboard or Management API, and wake deployments through the dashboard or Inference API. - [Manage the deployment lifecycle](https://docs.baseten.co/deployment/manage/lifecycle.md): Promote, deactivate, activate, and delete deployments from the dashboard, Baseten CLI, or Management API. - [Pull logs and metrics](https://docs.baseten.co/deployment/manage/logs-and-metrics.md): View logs and metrics in the dashboard, or fetch them with the Baseten CLI or Management API. - [Label deployments](https://docs.baseten.co/deployment/manage/labels.md): Add labels when you push a deployment, then view them in the dashboard, the Baseten CLI, or the Management API. #### Autoscaling - [Autoscaling](https://docs.baseten.co/deployment/autoscaling/overview.md): Configure autoscaling to dynamically adjust replicas based on traffic while minimizing idle compute costs. - [Request lifecycle](https://docs.baseten.co/deployment/autoscaling/request-lifecycle.md): How Dedicated Inference routes and queues requests, applies timeouts, and returns errors. - [Traffic patterns](https://docs.baseten.co/deployment/autoscaling/traffic-patterns.md): Identify your traffic pattern and configure autoscaling settings to match. - [Schedule autoscaling for traffic](https://docs.baseten.co/deployment/autoscaling/schedules.md): Change autoscaling settings on a recurring or one-time window instead of running an external cron job. - [Startup times](https://docs.baseten.co/deployment/autoscaling/startup-times.md): The phases of replica startup, how to measure startup time, and the model and config changes that reduce it. ### Engines - [Overview](https://docs.baseten.co/engines/index.md): Inference engines for embeddings, dense LLMs, MoE models, and Enterprise serving #### BEI - [Overview](https://docs.baseten.co/engines/bei/overview.md): Production-grade embeddings, reranking, and classification models - [BEI-Bert](https://docs.baseten.co/engines/bei/bei-bert.md): Bidirectional encoder embeddings with cold-start optimization - [Named entity recognition](https://docs.baseten.co/engines/bei/ner.md): Token-level entity classification on BEI-Bert with /predict_tokens #### Engine-Builder-LLM - [Overview](https://docs.baseten.co/engines/engine-builder-llm/overview.md): Dense LLM text generation with lookahead decoding and structured outputs - [Custom engine builder](https://docs.baseten.co/engines/engine-builder-llm/custom-engine-builder.md): Implement custom model.py for business logic, logging, and advanced inference patterns - [Speculative decoding](https://docs.baseten.co/engines/engine-builder-llm/lookahead-decoding.md): Lookahead decoding on Engine-Builder-LLM (v1) for code generation and predictable content - [LoRA support](https://docs.baseten.co/engines/engine-builder-llm/lora-support.md): Multi-LoRA adapters for Engine-Builder-LLM engine #### BIS-LLM - [Overview](https://docs.baseten.co/engines/bis-llm/overview.md): Token-based autoscaling, KV-aware routing, and speculative decoding for MoE and large dense models - [Migrate from Engine-Builder-LLM](https://docs.baseten.co/engines/bis-llm/migrate-from-v1.md): Translate a v1 Engine-Builder-LLM configuration to the v2 engine build, including the autoscaling, speculation, and routing changes that aren't just renames - [Advanced features for BIS-LLM](https://docs.baseten.co/engines/bis-llm/advanced-features.md): KV-aware routing, speculative decoding, and disaggregated serving #### Performance - [Quantization guide](https://docs.baseten.co/engines/performance-concepts/quantization-guide.md): FP8 and FP4 trade-offs and hardware requirements for all engines - [Autoscaling engines](https://docs.baseten.co/engines/performance-concepts/autoscaling-engines.md): Engine-specific autoscaling settings for BEI, Engine-Builder-LLM, and BIS-LLM - [Deploy from cloud storage](https://docs.baseten.co/engines/performance-concepts/cloud-storage-deployment.md): Connect your S3 bucket, GCS bucket, Azure container, or Hugging Face repository to Baseten's TRT-LLM inference engines and deploy without re-uploading weights. ### Model Labs - [Baseten for Model Labs](https://docs.baseten.co/labs/index.md): Control access to models through Frontier Gateway or distribute custom-trained models through Baseten. #### Distribution Platform - [Baseten Distribution Platform](https://docs.baseten.co/labs/platform/index.md): Sell a deployed model through Baseten using a shared Model API or customer-isolated Dedicated deployments. - [Manage your model](https://docs.baseten.co/labs/platform/manage.md): Deploy, validate, publish, update, and retire a model on the Distribution Platform. - [Report usage](https://docs.baseten.co/labs/platform/usage-reporting.md): Report billable usage for Distribution Platform inference so Baseten can bill customers and calculate your proceeds. - [Give customers access](https://docs.baseten.co/labs/platform/access.md): Give customers access to your Model API or Dedicated model and manage Model API rate limits. - [Insights](https://docs.baseten.co/labs/platform/insights.md): Review Distribution Platform models, customers, channels, and the displayed Est. gross proceeds (30d) field. ### Organization - [Organization settings](https://docs.baseten.co/organization/overview.md): Manage your Baseten organization's access, security, and resources. - [Billing and usage](https://docs.baseten.co/organization/billing.md): How Baseten meters per-minute usage, and how to manage payment, credits, and invoices for your workspace. - [Access control](https://docs.baseten.co/organization/access.md): Manage access to your Baseten organization with role-based access control. - [Teams](https://docs.baseten.co/organization/teams.md): Organize your organization into multiple teams with isolated resources and granular access control. - [Restricted environments](https://docs.baseten.co/organization/restricted-environments.md): Control access to sensitive environments like production with environment-level permissions. - [Audit logs](https://docs.baseten.co/organization/audit-logs.md): Track configuration and access changes across your Baseten organization, and export audit events to your SIEM. #### Authenticate to Baseten - [SSO and SCIM](https://docs.baseten.co/organization/sso-and-scim.md): Authenticate Baseten users through your identity provider and automatically provision accounts, directory groups, and roles. - [API keys](https://docs.baseten.co/organization/api-keys.md): Create, use, rotate, and revoke API keys for deployment, inference, and management. #### Access external resources - [Secrets](https://docs.baseten.co/organization/secrets.md): Store and access sensitive credentials in your deployed models. - [OpenID Connect (OIDC) authentication](https://docs.baseten.co/organization/oidc.md): Use short-lived OIDC tokens to securely authenticate to cloud resources - [Configure AWS AssumeRole authentication](https://docs.baseten.co/organization/aws-assume-role.md): Grant Baseten access to your AWS resources with an IAM trust policy ### Hosting options - [Choose your hosting option](https://docs.baseten.co/hosting-options/overview.md): Choose where your inference workloads run based on infrastructure ownership, isolation, connectivity, capacity, and data residency. - [Baseten Cloud](https://docs.baseten.co/hosting-options/cloud.md): Fully managed inference on Baseten's global fleet of elastic, on-demand compute. - [Single-tenant](https://docs.baseten.co/hosting-options/single-tenant.md): Dedicated, fully managed infrastructure with isolated compute and networking. - [Self-hosted](https://docs.baseten.co/hosting-options/self-hosted.md): Run high-scale inference on infrastructure you control. ### Observability - [Logs](https://docs.baseten.co/observability/logs.md): Scope logs by environment or deployment, then filter by request ID for individual predictions. - [Metrics](https://docs.baseten.co/observability/metrics.md): Compare request load, latency, and resource usage in the Metrics tab. - [Status and health](https://docs.baseten.co/observability/health.md): Interpret deployment statuses and use logs and metrics to investigate failures. - [Secure model inference](https://docs.baseten.co/observability/security.md): Data privacy, compliance policies, workload isolation, and self-hosted inference. - [Tracing](https://docs.baseten.co/observability/tracing.md): Export built-in and custom OpenTelemetry traces from a Truss deployment. #### Export metrics - [Export deployment metrics](https://docs.baseten.co/observability/export-metrics/overview.md): Scrape Baseten deployment metrics in Prometheus format and send them to your observability provider. - [Export custom metrics](https://docs.baseten.co/observability/export-metrics/custom-metrics.md): Send application-defined metrics from a Baseten deployment to your observability provider. - [Export to Prometheus](https://docs.baseten.co/observability/export-metrics/prometheus.md): Scrape Baseten metrics with Prometheus using a metrics-only API key. - [Export to Datadog](https://docs.baseten.co/observability/export-metrics/datadog.md): Scrape Baseten metrics with an OpenTelemetry Collector and send them to Datadog. - [Export to Grafana Cloud](https://docs.baseten.co/observability/export-metrics/grafana.md): Collect Baseten metrics with Grafana Cloud's hosted Metrics Endpoint integration. - [Export to New Relic](https://docs.baseten.co/observability/export-metrics/new-relic.md): Scrape Baseten metrics with an OpenTelemetry Collector and send them to New Relic over OTLP. ### Troubleshooting - [Deployments](https://docs.baseten.co/troubleshooting/deployments.md): Troubleshoot common problems during model deployment - [SSH access](https://docs.baseten.co/inference/ssh.md): Connect to running model deployments directly from your terminal with standard SSH. - [Inference](https://docs.baseten.co/troubleshooting/inference.md): Troubleshoot common problems during model inference - [Loops](https://docs.baseten.co/troubleshooting/loops.md): Resolve capacity, quota, and checkpoint errors and stop interrupted Loops runs. - [Inference errors](https://docs.baseten.co/inference/errors.md): What each inference error means and where to look next.