Skip to main content
Run Qwen 2.5 3B Instruct with vLLM on an L4 GPU. You’ll write a Truss configuration that connects the model weights, serving container, and hardware. vLLM loads the weights onto the GPU and serves an OpenAI-compatible API. Baseten runs the container and routes requests to it.

​
Set up your environment

You’ll need a Baseten account and uv for the Python client. This tutorial creates billable GPU compute. Check L4 pricing before deploying. Install the Baseten CLI:
Terminal
For other platforms or a specific version, see the Baseten CLI install reference. Open API keys, select Create API key, enter a name, select Personal, and create the key. Set it in your terminal:
The CLI and the Python client both read BASETEN_API_KEY. Run the remaining commands in this terminal so the key stays available to them.

​
Configure the model server

Create a project directory:
Terminal
Save the following as config.yaml. The vLLM container includes the inference server, so you don’t need to write a Dockerfile or Python server:
config.yaml
The configuration connects the components:
  • weights pins a Hugging Face revision and mounts its files at /models/qwen. This model is public and ungated, so the download doesn’t need a Hugging Face token.
  • base_image selects a versioned vLLM container. start_command tells vLLM to load /models/qwen and listen on port 8000, matching server_port.
  • --served-model-name sets the name in API requests. model_name sets the dashboard name.
  • The health endpoints let Baseten check whether the server is ready and running. predict_endpoint identifies the server route that handles inference.
  • resources assigns one L4 to each running copy of the server, called a replica. predict_concurrency lets Baseten forward up to 256 simultaneous requests to each replica so vLLM can batch them.
The openai-compatible tag enables the dashboard’s chat playground for this server. vLLM itself provides the OpenAI-compatible API.

​
Deploy the model

Run this command from the directory containing config.yaml:
--tail shows logs; --wait waits for Active or exits with an error. Baseten must load the container and weights before the server can answer requests. Copy the model ID (abc1d2ef) and deployment ID (xyz123) from the output. Use your own IDs in the request URL so you test the deployment you just created. The first published deployment of a new model serves production; later pushes need an explicit promotion to update production. If deployment fails, inspect the build or deployment logs. Build failures occur before the container starts; startup failures occur while the server or weights load. Use deployment troubleshooting to identify the failing stage.

​
Send a request

Save this OpenAI-compatible request as call_model.py. Replace {model_id} and {deployment_id} with the IDs from the push. The request’s model value matches --served-model-name in the configuration:
call_model.py
Run the script in the same terminal. --with openai installs the SDK for this run:
The response wording varies. The URL targets this deployment directly. To serve an update through the stable production endpoint, promote the validated deployment and verify the environment changed, then change the client’s base URL to https://model-{model_id}.api.baseten.co/environments/production/sync/v1 and run the request again.

​
Clean up

Exiting the client script doesn’t stop the deployment. When you’re finished testing, deactivate the deployment to release its replicas and stop its compute charges. Its endpoint stops serving requests; the model and configuration remain available for reactivation.
  1. Open Dedicated Inference in the dashboard and select the model you created.
  2. Under Deployments, select the deployment you used in this example.
  3. Select Deactivate deployment, then Yes, deactivate.
  4. Wait for the deployment’s status to become Inactive.
See Manage the deployment lifecycle to reactivate it later.

​
Next steps