Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.baseten.co/llms.txt

Use this file to discover all available pages before exploring further.

When you need custom preprocessing, postprocessing, or want to run a model that isn’t supported by Baseten’s built-in engines, you can write Python code in a model.py file. Truss provides a Model class with three methods (__init__, load, and predict) that give you full control over how your model initializes, loads weights, and handles requests. Most deployments don’t need custom Python at all. If you’re deploying a supported open-source model, see Build your first model for the config-only approach. Use custom model code when you need to:
  • Run a model architecture that Baseten’s engines don’t support.
  • Add custom preprocessing or postprocessing around inference.
  • Combine multiple models or libraries in a single endpoint.

​
Prerequisites

You need uv installed and a Baseten account with an API key.

​
Initialize your model

Create a new Truss project with truss init. 
$ truss init hello-world
? 📦 Name this model: HelloWorld
Truss HelloWorld was created in ~/hello-world
This creates a directory with the following structure:
  • config.yaml: Configuration for dependencies, resources, and deployment settings.
  • model/model.py: Your model code.
  • packages/: Optional local Python packages.
  • data/: Optional data files bundled with your model.

​
config.yaml

The config.yaml file configures dependencies, resources, and other settings. Here’s the default:
config.yaml
# yaml-language-server: $schema=https://raw.githubusercontent.com/basetenlabs/truss/main/truss/config.schema.json
model_name: HelloWorld
python_version: py313
resources:
  accelerator: null
  cpu: '1'
  memory: 2Gi
  use_gpu: false
truss init writes python_version based on your local Python (3.9 through 3.13). use_gpu is computed from accelerator; setting it manually has no effect.
The fields you’ll use most often:
  • requirements: Python packages installed at build time (pip format).
  • resources: CPU, memory, and GPU allocation.
  • secrets: Secret names your model needs at runtime, such as Hugging Face API keys.
See the Configuration page for the full reference.

​
model.py

The model.py file defines a Model class with three methods: 
class Model:
    def __init__(self, **kwargs):
        pass

    def load(self):
        pass

    def predict(self, model_input):
        return model_input
  • __init__: Runs when the class is created. Initialize variables and store configuration here.
  • load: Runs once at startup, before any requests. Load model weights, tokenizers, and other heavy resources here. Separating this from __init__ keeps expensive operations out of the request path.
  • predict: Runs on every API request. Process input, run inference, and return the response.
load and predict don’t run on the same thread, which matters for GPU workloads where state can be tied to the creating thread (such as CUDA contexts). With sync predict and the default predict_concurrency of 1, successive predict calls often reuse the same worker thread, but Baseten doesn’t guarantee it.

​
Give your model access to files

Most models need additional files at runtime, such as weights, tokenizers, configs, or reference datasets. For local files under ~1 GB total, bundle them in your Truss’s data/ directory and access them in __init__ through kwargs["data_dir"]:
model.py
class Model:
    def __init__(self, **kwargs):
        self._data_dir = kwargs["data_dir"]

    def load(self):
        self.tokenizer = AutoTokenizer.from_pretrained(str(self._data_dir))
For larger weights or remote sources (Hugging Face, S3, GCS, R2), use the Baseten Delivery Network (BDN) instead; it mirrors weights once and serves them from caches close to your pods. For all options, see Data and storage.

​
Deploy your model

Deploy with truss push --watch. 
$ truss push --watch
This packages your code and config, builds a container, and deploys it to Baseten.

​
Invoke your model

After deployment, call your model at the invocation URL: 
$ curl -X POST https://model-{model-id}.api.baseten.co/development/predict \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '"some text"'
You should see: 
"some text"

​
Example: text classification

To see the Model class in action, deploy a text classification model from Hugging Face using the transformers library.

​
Update config.yaml

Add transformers and torch as dependencies:
config.yaml
requirements:
  - transformers
  - torch

​
Update model.py

Load the classification pipeline in load and run it in predict:
model.py
from transformers import pipeline

class Model:
    def __init__(self, **kwargs):
        pass

    def load(self):
        self._model = pipeline("text-classification")

    def predict(self, model_input):
        return self._model(model_input)

​
Deploy and call

Deploy with truss push --watch, then call the endpoint: 
$ truss push --watch

$ curl -X POST https://model-{model-id}.api.baseten.co/development/predict \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '"some text"'

​
Next steps

  • Implementation: Core Model lifecycle, method signatures, and sync vs. async predict patterns.
  • HTTP endpoints: Add chat_completions, completions, embeddings, messages, or responses to serve matching /v1/* routes from custom model code.
  • Streaming output: Return generated output incrementally instead of waiting for the full response.
  • Custom health checks: Define readiness and liveness behavior for custom model logic.
  • Configuration: Full reference for config.yaml options.
  • Build your first model: Deploy a model with just a config file, no custom Python needed.