Skip to main content
To deploy Qwen 2.5 3B Instruct on an L4 with an OpenAI-compatible API, write a config.yaml and run truss push. You need a Baseten account and uv.

Set up your environment

To install Truss and sign in:
1

Install Truss

Terminal
To skip a permanent install, run uvx truss … for the rest of this page.
2

Sign in

Browser login opens a tab to approve this device, so you don’t copy an API key yet:
Terminal
If the browser doesn’t open, copy the URL from the terminal and approve the device there.

Create the config

To create the project:
  1. Create a directory:
    Terminal
  2. Add a config.yaml:
    config.yaml
These fields control the result:
  • resources.accelerator: L4 runs inference on a single L4 (24 GB VRAM).
  • trt_llm compiles the model with TensorRT-LLM.
  • checkpoint_repository points to weights on Hugging Face. Qwen 2.5 3B Instruct is ungated, so you don’t need a token.
  • quantization_type: fp8 halves weight memory by quantizing to 8-bit floats.
  • num_builder_gpus: 1 sets the GPU count for the engine-build job. Without it, the CLI warns that FP8 builds can OOM at build time.

Deploy the model

To push the model:
  1. From the project directory, run:
    Terminal
    You should see:
  2. Open the logs URL. Baseten downloads the weights, compiles them with TensorRT-LLM, and deploys the container to an L4. Compilation can take several minutes. When the dashboard shows Active, the endpoint can serve requests.
If truss push warns that the FP8 build can OOM, confirm num_builder_gpus: 1 is in config.yaml and push again.

Call the model

Engine-based deployments serve an OpenAI-compatible API. Replace {model_id} with the model ID from the push output. The endpoint follows this shape: Anatomy of the model API endpoint. In https://model-abc123.api.baseten.co/environments/production/sync/v1, abc123 is the model ID and production is the environment that serves the request. To send a request:
  1. Create a personal API key and export it. Browser login does not set this variable:
  2. Call the endpoint. In Python, uv run --with openai installs the SDK for that run:
call_model.py
Terminal
You should see a response like:
If the request returns 401, confirm BASETEN_API_KEY is set in the same shell. If it returns 404, replace {model_id} with the ID from the push output and wait until the dashboard shows Active.

Next steps

Engine configuration

Tune max sequence length, batch size, quantization, and runtime settings for your deployment.

Custom model code

Add custom Python when you need preprocessing, postprocessing, or unsupported model architectures.

Custom Docker server

Run vLLM, SGLang, Triton, or any containerized inference server on Baseten.

Autoscaling

Configure replicas, concurrency targets, and scale-to-zero for production traffic.

Promote to production

Move from development to production with truss push --promote.