Deploy your first model

Deploy Qwen 2.5 3B Instruct on an L4 GPU and send an inference request. The Baseten CLI submits your configuration to Engine-Builder-LLM, which compiles the model with TensorRT-LLM and creates a dedicated endpoint. You’ll need a Baseten account and uv for the Python client. This example creates billable GPU compute. Check L4 pricing before deploying.

macOS or Linux
Windows

Terminal

brew tap basetenlabs/baseten
brew trust --formula basetenlabs/baseten/baseten
brew install baseten

Download and extract the Windows archive for the latest Baseten CLI release:

Terminal

$release = Invoke-RestMethod https://api.github.com/repos/basetenlabs/baseten-cli/releases/latest
$asset = $release.assets | Where-Object { $_.name -like "*_windows_amd64.zip" }
Invoke-WebRequest $asset.browser_download_url -OutFile baseten.zip
Expand-Archive -Force baseten.zip .

Move baseten.exe to a directory on your PATH.

For other platforms or a specific version, see the Baseten CLI install reference. Sign in with the browser:

Terminal

baseten auth login

Configure the model

Create a project directory:

mkdir qwen-2.5-3b && cd qwen-2.5-3b

Create config.yaml in this directory:

config.yaml

model_metadata:
  tags:
    - openai-compatible
model_name: Qwen-2.5-3B
resources:
  accelerator: L4
  use_gpu: true
trt_llm:
  build:
    base_model: decoder
    checkpoint_repository:
      source: HF
      repo: "Qwen/Qwen2.5-3B-Instruct"
    max_seq_len: 8192
    quantization_type: fp8
    tensor_parallel_count: 1
    num_builder_gpus: 2

The configuration selects Qwen’s public weights, an L4 GPU, and FP8 quantization. Engine-Builder-LLM compiles the model using the builder GPUs, then serves it on the L4. See the engine configuration reference for these settings.

Deploy the model

Run this command from the directory containing config.yaml:

baseten model push --tail --wait

  Model:       Qwen-2.5-3B (abc1d2ef)
  Deployment:  xyz123
  Environment: production

--tail shows logs; --wait waits for Active. Baseten must finish compiling and loading the model before it can answer requests. Save your model ID from the output, abc1d2ef in this example.

Call the model

Engine-based deployments serve an OpenAI-compatible API. Once the deployment becomes Active, create a personal key in API keys, or reuse one you’ve saved. Set it in the terminal where you’ll run the request:

export BASETEN_API_KEY="YOUR_API_KEY"

$env:BASETEN_API_KEY = "YOUR_API_KEY"

Replace {model_id} with the model ID from the deployment output. Save this code as chat.py:

chat.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="Qwen-2.5-3B",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

Run it in the same terminal. --with openai installs the SDK for this run:

uv run --with openai chat.py

Machine learning is a branch of artificial intelligence where systems learn
patterns from data to make predictions or decisions without being explicitly
programmed for each task...

The response wording varies. The URL sends the request to your model’s production environment.

Clean up

Exiting the client script doesn’t stop the deployment. When you’re finished testing, deactivate the deployment to release its replicas and stop its compute charges. Its endpoint stops serving requests; the model and configuration remain available for reactivation.

Open Dedicated Inference in the dashboard and select the model you created.
Under Deployments, select the deployment you used in this example.
Select Deactivate deployment, then Yes, deactivate.
Wait for the deployment’s status to become Inactive.

See Manage the deployment lifecycle to reactivate it later.

Next steps

Configure autoscaling to control replica counts and idle compute cost.
Configure a model server to use your own serving container.
Deploy a model through Explore or the Management API.

Examples

Models

Engines

Custom Docker servers

Custom Python models

Chains

Post-training

Baseten for Model Labs

Deploy your first model

Configure the model

Deploy the model

Call the model

Clean up

Next steps

​Install and sign in

​Configure the model

​Deploy the model

​Call the model

​Clean up

​Next steps

Install and sign in

Configure the model

Deploy the model

Call the model

Clean up

Next steps