Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.baseten.co/llms.txt

Use this file to discover all available pages before exploring further.

SGLang is a high-performance serving framework for LLMs that supports a wide range of models and optimization techniques. This guide deploys an SGLang model as a custom Docker server on Baseten. This configuration serves Qwen 2.5 3B with SGLang on an L4 GPU. The deployment process is the same for larger models like GLM-4.7. Adjust the resources and start_command to match your model’s requirements.

​
Set up your environment

This guide uses uvx to run Truss commands without a separate install step. Sign in to Baseten and install the OpenAI SDK. Browser login opens a tab to approve this device, so there’s no API key to copy and paste.
Sign in to Baseten
uvx truss login --browser
Install the OpenAI SDK
uv pip install openai
Hugging Face access for gated models. Some models require that you accept terms and conditions on Hugging Face before deployment. To prevent issues:
  1. Accept the license for any gated models you wish to access, like Gemma 3.
  2. Create a read-only user access token from your Hugging Face account.
  3. Add the hf_access_token secret to your Baseten workspace.

​
Configure the model

Create a directory with a config.yaml file: 
mkdir qwen-2-5-3b-sglang
touch qwen-2-5-3b-sglang/config.yaml
Copy the following configuration into config.yaml:
config.yaml
model_metadata:
  example_model_input:
    messages:
      - role: system
        content: "You are a helpful assistant."
      - role: user
        content: "What does Tongyi Qianwen mean?"
    stream: true
    model: Qwen/Qwen2.5-3B-Instruct
    max_tokens: 512
    temperature: 0.6
  tags:
    - openai-compatible
model_name: Qwen 2.5 3B SGLang
base_image:
  image: lmsysorg/sglang:v0.5.8.post1
docker_server:
  start_command: python3 -m sglang.launch_server --model-path /models/qwen --served-model-name Qwen/Qwen2.5-3B-Instruct --host 0.0.0.0 --port 8000
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000
weights:
  - source: "hf://Qwen/Qwen2.5-3B-Instruct@aa8e72537993ba99e69dfaafa59ed015b17504d1"
    mount_location: "/models/qwen"
resources:
  accelerator: L4
  use_gpu: true
runtime:
  predict_concurrency: 32
  health_checks:
    restart_threshold_seconds: 300
    stop_traffic_threshold_seconds: 120
The base_image specifies the SGLang Docker image. The weights block uses the Baseten Delivery Network to mirror the model from Hugging Face and mount it at /models/qwen before the container starts. SGLang reads weights directly from that path and serves the model with --served-model-name, which sets the model identifier for the OpenAI-compatible API. The readiness_endpoint and liveness_endpoint use /health, which returns 200 once the server is running. The health_checks settings control how Baseten monitors the server after it passes the startup probe.

​
Deploy the model

Push the model to Baseten to start the deployment: 
uvx truss push qwen-2-5-3b-sglang
You should see output like: 
Deploying truss using L4:4x16 instance type.
Model Qwen 2.5 3B SGLang was successfully pushed.
View logs at https://app.baseten.co/models/XXXXXXX/logs/XXXXXXX
Copy the model URL from the output for the next step. The first deploy can take several minutes while Baseten pulls the SGLang base image (around 18 GB). Subsequent scale-ups reuse the cached image and start much faster.

​
Call the model

Call the deployed model with the OpenAI client:
call_model.py
import os
from openai import OpenAI

model_url = "https://model-XXXXXXX.api.baseten.co/environments/production/sync/v1"

client = OpenAI(
    base_url=model_url,
    api_key=os.environ.get("BASETEN_API_KEY"),
)

stream = client.chat.completions.create(
    model="Qwen/Qwen2.5-3B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What does Tongyi Qianwen mean?"}
    ],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="")
Replace the model_url with the URL from your deployment output.