Skip to main content
Deploy training checkpoints and cloud storage models with Engine-Builder-LLM, BEI, or BIS-LLM.

​
Training checkpoint deployment

Deploy fine-tuned models from Baseten Training with Engine-Builder-LLM. Specify BASETEN_TRAINING as the source:
config.yaml
model_name: My Fine-Tuned LLM
resources:
  accelerator: H100:1
  use_gpu: true
secrets:
  hf_access_token: null # do not set value here
trt_llm:
  build:
    base_model: decoder
    checkpoint_repository:
      source: BASETEN_TRAINING
      repo: YOUR_TRAINING_JOB_ID
      revision: checkpoint-100
Key fields:
  • base_model: decoder for LLMs, encoder/encoder_bert for embeddings
  • source: BASETEN_TRAINING for Baseten Training checkpoints
  • repo: Your training job ID
  • revision: Checkpoint folder name (e.g., checkpoint-100, checkpoint-final)
Find your checkpoint details with: 
truss train get_checkpoint_urls --job-id=YOUR_TRAINING_JOB_ID

​
Encoder model requirements

To deploy a fine-tuned encoder model (embeddings, rerankers) from training checkpoints, use encoder or encoder_bert as the base model:
config.yaml
model_name: My Fine-Tuned Embeddings
resources:
  accelerator: L4:1
  use_gpu: true
trt_llm:
  build:
    base_model: encoder_bert
    checkpoint_repository:
      source: BASETEN_TRAINING
      repo: YOUR_TRAINING_JOB_ID
      revision: checkpoint-final
  runtime:
    webserver_default_route: /v1/embeddings
Use encoder_bert for BERT-based models (sentence-transformers, classification, reranking). Use encoder for causal embedding models. Encoder models have specific requirements:
  • No tensor parallelism: Omit tensor_parallel_count or set it to 1.
  • Fast tokenizer required: Your checkpoint must include a tokenizer.json file. Models using only the legacy vocab.txt format are not supported.
  • Embedding model files: For sentence-transformer models, include modules.json and 1_Pooling/config.json in your checkpoint.
The webserver_default_route configures the inference endpoint. Options include /v1/embeddings for embeddings, /rerank for rerankers, and /predict for classification.

​
Cloud storage deployment

Deploy models directly from S3, GCS, or Azure. Specify the storage source and bucket path:
config.yaml
trt_llm:
  build:
    base_model: decoder
    checkpoint_repository:
      source: S3  # or GCS, AZURE, HF
      repo: s3://your-bucket/path/to/model/
Storage sources:
  • S3: Amazon S3 buckets
  • GCS: Google Cloud Storage
  • AZURE: Azure Blob Storage
  • HF: Hugging Face repositories

​
Private storage setup

All runtimes use the same downloader system as model_cache. As a result, you configure the runtime_secret_name and repo identically across model_cache and runtimes like Engine-Builder-LLM or BEI. Secret Setup: Add these JSON secrets to your Baseten secrets manager. For more details, refer to the documentation in model_cache. S3: 
{
  "access_key_id": "XXXXX",
  "secret_access_key": "xxxxx/xxxxxx",
  "region": "us-west-2"
}
GCS: 
{
  "private_key_id": "xxxxxxx",
  "private_key": "-----BEGIN PRIVATE KEY-----\nMI",
  "client_email": "[email protected]"
}
Azure: 
{
  "account_key": "xxxxx"
}
Reference the secret in your config: 
secrets:
  aws_secret_json: "set token in baseten workspace"
trt_llm:
  build:
    checkpoint_repository:
      source: S3
      repo: s3://your-private-bucket/model
      runtime_secret_name: aws_secret_json
For Baseten Training deployments: These secrets are automatically mounted and available to your deployment.

​
Further reading