Skip to main content
Deploy Qwen 2.5 3B Instruct on an L4 with vLLM’s OpenAI-compatible API. You’ll define the model in config.yaml, then use the Baseten CLI to deploy and manage it. You need a Baseten account. To call the model, use cURL or run the Python example with uv.

Set up your environment

To install the Baseten CLI and sign in:
1

Install the Baseten CLI

Terminal
For Windows or a manual install, see the Baseten CLI installation guide.
2

Sign in

Browser login opens a tab where you approve the device. You do not need an API key until you call the deployed model:
Terminal
If the browser doesn’t open, copy the URL from the terminal and approve the device there.

Create the config

To create the project:
  1. Create a directory:
    Terminal
  2. Add a config.yaml:
    config.yaml
The key fields are:
  • model_metadata.tags: Marks the server as OpenAI-compatible, which enables the chat playground for this model in the dashboard.
  • base_image.image: Uses a pinned vLLM Docker image.
  • docker_server.start_command: Starts vLLM with the mounted weights. --served-model-name sets the model identifier in requests, and --port 8000 must match server_port.
  • docker_server.predict_endpoint: Sets vLLM’s /v1/chat/completions route as the deployment’s default request endpoint.
  • weights: Mounts the pinned Hugging Face weights at /models/qwen through the Baseten Delivery Network. This model is ungated, so it doesn’t need a token.
  • resources.accelerator: Runs inference on a single L4.
  • runtime.predict_concurrency: Allows up to 256 concurrent requests so vLLM can batch them.
For all available fields, see the Truss configuration reference.

Deploy the model

From the project directory, run:
Terminal
You should see:
The command streams deployment logs and exits when the model is Active.

Call the model

vLLM serves an OpenAI-compatible API. Replace {model_id} with the model ID from the push output. The endpoint follows this shape: Anatomy of the model API endpoint. In https://model-abc123.api.baseten.co/environments/production/sync/v1, abc123 is the model ID and production is the environment that serves the request. To send a request:
  1. Create a personal API key and export it. Browser login does not set this variable:
  2. Call the endpoint. The model value must match --served-model-name in the start_command. For the Python example, uv run --with openai installs the OpenAI SDK for that run:
call_model.py
Terminal
You should see a response like:
If the request fails, see Inference errors. You configured vLLM to serve a pinned Hugging Face model, deployed it to an L4 with the Baseten CLI, and called its OpenAI-compatible endpoint.

Choose your next learning path

Operate your model in production

  1. Use deployment environments to control which deployment serves production traffic.
  2. Configure autoscaling for your traffic patterns.
  3. Monitor the deployment with logs and metrics.

Customize the serving stack

  1. Explore model deployment examples to deploy other models and serving engines.
  2. Follow the vLLM recipe to tune the server and add streaming.
  3. Learn how to deploy custom Docker servers.
  4. Configure weight mounting for private repositories or cloud storage.

Add custom model logic

  1. Implement the Model class for preprocessing, postprocessing, or other architectures.
  2. Add Python dependencies required by your model code.
  3. Deploy and iterate as you develop locally.