Build your model

​

Set up your environment

brew tap basetenlabs/baseten
brew install baseten

baseten auth login --web

​

Create the config

1. Create a directory:
   Terminal

   mkdir qwen-2.5-3b && cd qwen-2.5-3b
   
2. Add a config.yaml:
   config.yaml

   model_name: Qwen-2.5-3B
   model_metadata:
     tags:
       - openai-compatible
   base_image:
     image: vllm/vllm-openai:v0.27.1
   docker_server:
     start_command: vllm serve /models/qwen --served-model-name Qwen/Qwen2.5-3B-Instruct --host 0.0.0.0 --port 8000
     readiness_endpoint: /health
     liveness_endpoint: /health
     predict_endpoint: /v1/chat/completions
     server_port: 8000
   weights:
     - source: "hf://Qwen/Qwen2.5-3B-Instruct@aa8e72537993ba99e69dfaafa59ed015b17504d1"
       mount_location: /models/qwen
   resources:
     accelerator: L4
     use_gpu: true
   runtime:
     predict_concurrency: 256
   

• model_metadata.tags: Marks the server as OpenAI-compatible, which enables the chat playground for this model in the dashboard.
• base_image.image: Uses a pinned vLLM Docker image.
• docker_server.start_command: Starts vLLM with the mounted weights. --served-model-name sets the model identifier in requests, and --port 8000 must match server_port.
• docker_server.predict_endpoint: Sets vLLM’s /v1/chat/completions route as the deployment’s default request endpoint.
• weights: Mounts the pinned Hugging Face weights at /models/qwen through the Baseten Delivery Network. This model is ungated, so it doesn’t need a token.
• resources.accelerator: Runs inference on a single L4.
• runtime.predict_concurrency: Allows up to 256 concurrent requests so vLLM can batch them.

​

Deploy the model

baseten model push --tail --wait

✨ Model Qwen-2.5-3B was successfully pushed ✨

  Model:       Qwen-2.5-3B (abc123)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc123 --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc123 --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc123/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc123.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc123

​

Call the model

1. Create a personal API key and export it. Browser login does not set this variable:
   macOS/Linux

   Windows

   export BASETEN_API_KEY="paste-your-api-key-here"
   
2. Call the endpoint. The model value must match --served-model-name in the start_command. For the Python example, uv run --with openai installs the OpenAI SDK for that run:

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-3B-Instruct",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

uv run --with openai python call_model.py

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "Qwen/Qwen2.5-3B-Instruct",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

Machine learning uses algorithms to learn patterns from data and make
predictions or decisions without being explicitly programmed for each task...

​

Choose your next learning path

​

Operate your model in production

1. Use deployment environments to control which deployment serves production traffic.
2. Configure autoscaling for your traffic patterns.
3. Monitor the deployment with logs and metrics.

​

Customize the serving stack

1. Explore model deployment examples to deploy other models and serving engines.
2. Follow the vLLM recipe to tune the server and add streaming.
3. Learn how to deploy custom Docker servers.
4. Configure weight mounting for private repositories or cloud storage.

​

Add custom model logic

1. Implement the Model class for preprocessing, postprocessing, or other architectures.
2. Add Python dependencies required by your model code.
3. Deploy and iterate as you develop locally.