Qwen3.8

Reasoning Tool calling Multimodal (image)Long context Agentic

Dense 27B-parameter multimodal reasoning model with FP8 weights, served on a single H100 with vLLM, MTP speculative decoding, and a data-parallel vision encoder.

Setup

Sign in to Baseten

uvx truss login --browser

Install the OpenAI SDK

uv pip install openai

This preset serves Qwen3.8-27B on H100:1 with block-scaled FP8 weights, optimized for low-latency interactive chat, reasoning, and agent workflows.

Hardware

H100 × 1

Engine

vLLM (4a2f33a884222f70… build)

Context

256K

Concurrency

Write the config

Create and move into the project directory:

mkdir qwen3.8-27b-latency && cd qwen3.8-27b-latency

Then create a file named config.yaml and paste the following:

config.yaml

model_name: "model:qwen3.8-27b preset:latency"

model_metadata:
  description: >-
    Qwen3.8-27B FP8 is a 27B dense multimodal reasoning model served through vLLM with
    OpenAI-compatible chat completions, tool calling, MTP speculative decoding, a
    data-parallel vision encoder, and CPU KV-cache offload.
  repo_id: Qwen/Qwen3.8-27B-FP8
  example_model_input:
    model: "Qwen/Qwen3.8-27B"
    messages:
      - role: user
        content: "Give me three prime numbers greater than 100."
    stream: true
    max_tokens: 512
    temperature: 1.0
    top_p: 0.95
  tags:
    - openai-compatible
    - multimodal
    - reasoning
    - tool-use

base_image:
  # qwen38 includes unreleased gated-delta-network MTP fixes required by this recipe.
  image: "vllm/vllm-openai:qwen38@sha256:4a2f33a884222f7049b983263ad9976f89452bb81affecf5b67d89ad35c1bc31"

weights:
  - source: "hf://Qwen/Qwen3.8-27B-FP8@017b9c7af6b5689d5dd426a76e0bc077eb5ca20a"
    mount_location: "/app/checkpoint/model"
    auth_secret_name: "hf_access_token"

secrets:
  hf_access_token: null

environment_variables:
  HF_HUB_ENABLE_HF_TRANSFER: "1"
  VLLM_LOGGING_LEVEL: WARNING
  VLLM_ENGINE_READY_TIMEOUT_S: "3600"

docker_server:
  start_command: >-
    vllm serve /app/checkpoint/model
    --served-model-name Qwen/Qwen3.8-27B
    --host 0.0.0.0
    --port 8000
    --tensor-parallel-size 1
    --max-model-len 262144
    --kv-cache-dtype fp8
    --gpu-memory-utilization 0.90
    --max-num-seqs 64
    --enable-prefix-caching
    --limit-mm-per-prompt.image 1
    --mm-encoder-tp-mode data
    --reasoning-parser qwen3
    --enable-auto-tool-choice
    --tool-call-parser qwen3_coder
    --speculative-config.method mtp
    --speculative-config.num_speculative_tokens 3
    --kv-transfer-config.kv_connector SimpleCPUOffloadConnector
    --kv-transfer-config.kv_role kv_both
    --kv-transfer-config.kv_connector_extra_config.cpu_bytes_to_use_per_rank 68719476736
    --kv-transfer-config.kv_connector_extra_config.lazy_offload false
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000

resources:
  accelerator: H100:1
  node_count: 1
  cpu: "16"
  memory: 118Gi
  use_gpu: true

runtime:
  predict_concurrency: 64
  health_checks:
    restart_check_delay_seconds: 1800
    restart_threshold_seconds: 1200
    stop_traffic_threshold_seconds: 120

This preset deploys the official FP8 checkpoint of Qwen3.8-27B through vLLM on a single H100. It enables MTP speculative decoding with three speculative tokens and CPU KV-cache offload, keeping time-to-first-token low while fitting the 262,144-token context window in a single GPU’s memory. The deployment exposes an OpenAI-compatible chat completions endpoint with reasoning, tool calling, and one image per prompt.

Flags

The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:

Flag	Value	What it does
`--tensor-parallel-size`	`1`	Number of GPUs to shard the model across.
`--max-model-len`	`262144`	Maximum context length (tokens) the server accepts per request.
`--kv-cache-dtype`	`fp8`	KV cache numeric precision. fp8: ~2× KV cache density with negligible quality impact on most models.
`--gpu-memory-utilization`	`0.90`	Fraction of GPU memory vLLM may use for weights and KV cache.
`--max-num-seqs`	`64`	Maximum number of concurrent sequences in the batch.
`--enable-prefix-caching`	(no value)	Reuse KV cache across requests that share a prefix.
`--limit-mm-per-prompt.image`	`1`	Maximum number of image inputs per prompt.
`--mm-encoder-tp-mode`	`data`	Tensor-parallel mode for the multimodal vision encoder. data: Data-parallel encoding; each rank processes different images independently.
`--reasoning-parser`	`qwen3`	Server-side parser that separates reasoning output into `reasoning_content`. qwen3: Qwen3-family thinking format (used by Qwen3, Qwen3.5, and Qwen3.6).
`--enable-auto-tool-choice`	(no value)	Let the model choose when to call tools without requiring `tool_choice: "required"`.
`--tool-call-parser`	`qwen3_coder`	Server-side parser that emits structured `tool_calls` on the response. qwen3_coder: Qwen3-Coder tool format.
`--speculative-config.method`	`mtp`	Speculative decoding method. mtp: Multi-token prediction head speculation.
`--speculative-config.num_speculative_tokens`	`3`	Number of tokens the draft speculator proposes per step.
`--kv-transfer-config.kv_connector`	`SimpleCPUOffloadConnector`	KV cache transfer connector for offloading cache to host memory or another node. SimpleCPUOffloadConnector: Offloads KV cache to CPU memory to extend the effective KV cache capacity beyond GPU memory.
`--kv-transfer-config.kv_role`	`kv_both`	Role this node plays in KV cache transfer. kv_both: Both sender and receiver, used for single-node CPU offload.
`--kv-transfer-config.kv_connector_extra_config.cpu_bytes_to_use_per_rank`	`68719476736`	Bytes of CPU memory allocated per rank for KV cache offload.
`--kv-transfer-config.kv_connector_extra_config.lazy_offload`	`false`	Whether to offload KV cache lazily (on demand) or eagerly. false: Eager offload; KV cache is copied to CPU memory as soon as it is produced.

Deploy

Push the config to Baseten:

uvx truss push

You should see output similar to:

✨ Model qwen3.8-27b-latency was successfully pushed ✨

   Model ID:      abc1d2ef
   Deployment ID: xyz123
   Endpoint:      model-abc1d2ef.api.baseten.co
   Logs:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

truss push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.

Call the model

Your deployment serves an OpenAI-compatible API. Now call your deployment to run inference:

Python
cURL

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.8-27B",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "Qwen/Qwen3.8-27B",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

To access the model’s chain of thought, enable thinking mode. The server parses the reasoning output into a separate reasoning_content field on the response:

response = client.chat.completions.create(
    model="Qwen/Qwen3.8-27B",
    messages=[
        {"role": "user", "content": "How many r's in strawberry?"}
    ],
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
print(response.choices[0].message.reasoning_content)  # chain of thought
print(response.choices[0].message.content)            # final answer

To let the model call tools, pass a tools array. The server returns structured tool_calls on the response:

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "parameters": {
            "type": "object",
            "properties": {"location": {"type": "string"}},
            "required": ["location"],
        },
    },
}]

response = client.chat.completions.create(
    model="Qwen/Qwen3.8-27B",
    messages=[
        {"role": "user", "content": "What's the weather in Paris?"}
    ],
    tools=tools,
)
print(response.choices[0].message.tool_calls)

Examples

Models

Engines

Custom Docker servers

Custom Python models

Chains

Setup

Hardware

Engine

Context

Concurrency

Write the config

Flags

Deploy

Call the model

Next steps

Call your model

Autoscaling

​Setup

Hardware

Engine

Context

Concurrency

​Write the config

​Flags

​Deploy

​Call the model

​Next steps

Call your model

Autoscaling

Setup

Write the config

Flags

Deploy

Call the model

Next steps