Build your model

​

Set up your environment

uv tool install truss

truss login --browser

​

Create the config

1. Create a directory:
   Terminal

   mkdir qwen-2.5-3b && cd qwen-2.5-3b
   
2. Add a config.yaml:
   config.yaml

   model_name: Qwen-2.5-3B
   resources:
     accelerator: L4
   model_metadata:
     tags:
       - openai-compatible
   trt_llm:
     build:
       base_model: decoder
       checkpoint_repository:
         source: HF
         repo: "Qwen/Qwen2.5-3B-Instruct"
       max_seq_len: 8192
       quantization_type: fp8
       tensor_parallel_count: 1
       num_builder_gpus: 1
   

• resources.accelerator: L4 runs inference on a single L4 (24 GB VRAM).
• trt_llm compiles the model with TensorRT-LLM.
• checkpoint_repository points to weights on Hugging Face. Qwen 2.5 3B Instruct is ungated, so you don’t need a token.
• quantization_type: fp8 halves weight memory by quantizing to 8-bit floats.
• num_builder_gpus: 1 sets the GPU count for the engine-build job. Without it, the CLI warns that FP8 builds can OOM at build time.

​

Deploy the model

1. From the project directory, run:
   Terminal

   truss push
   
   You should see:
   ✨ Model Qwen-2.5-3B was successfully pushed ✨
   
      Model ID:      abc1d2ef
      Deployment ID: xyz123
      Endpoint:      model-abc1d2ef.api.baseten.co
      Logs:          https://app.baseten.co/models/abc1d2ef/logs/xyz123
   
2. Open the logs URL. Baseten downloads the weights, compiles them with TensorRT-LLM, and deploys the container to an L4. Compilation can take several minutes. When the dashboard shows Active, the endpoint can serve requests.

​

Call the model

1. Create a personal API key and export it. Browser login does not set this variable:
   macOS/Linux

   Windows

   export BASETEN_API_KEY="paste-your-api-key-here"
   
2. Call the endpoint. In Python, uv run --with openai installs the SDK for that run:

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="Qwen-2.5-3B",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

uv run --with openai python call_model.py

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "Qwen-2.5-3B",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

Machine learning is a branch of artificial intelligence where systems learn
patterns from data to make predictions or decisions without being explicitly
programmed for each task...

​

Next steps