> ## Documentation Index > Fetch the complete documentation index at: https://docs.baseten.co/llms.txt > Use this file to discover all available pages before exploring further. # Qwen3.8 > Dense 27B-parameter multimodal reasoning model with FP8 weights, served on a single H100 with vLLM, MTP speculative decoding, and a data-parallel vision encoder.
Reasoning Tool calling Multimodal (image) Long context Agentic
Dense 27B-parameter multimodal reasoning model with FP8 weights, served on a single H100 with vLLM, MTP speculative decoding, and a data-parallel vision encoder. ## Setup Sign in to Baseten with Truss, then install the OpenAI SDK. **Sign in to Baseten** ```sh theme={"system"} uvx truss login --browser ``` **Install the OpenAI SDK** ```sh theme={"system"} uv pip install openai ``` This preset serves Qwen3.8-27B on H100:1 with block-scaled FP8 weights, optimized for low-latency interactive chat, reasoning, and agent workflows. H100 × 1 vLLM (4a2f33a884222f70... build) 256K 64 ## Write the config Create and move into the project directory: ```sh theme={"system"} mkdir qwen3.8-27b-latency && cd qwen3.8-27b-latency ``` Then create a file named `config.yaml` and paste the following: ```yaml config.yaml theme={"system"} model_name: "model:qwen3.8-27b preset:latency" model_metadata: description: >- Qwen3.8-27B FP8 is a 27B dense multimodal reasoning model served through vLLM with OpenAI-compatible chat completions, tool calling, MTP speculative decoding, a data-parallel vision encoder, and CPU KV-cache offload. repo_id: Qwen/Qwen3.8-27B-FP8 example_model_input: model: "Qwen/Qwen3.8-27B" messages: - role: user content: "Give me three prime numbers greater than 100." stream: true max_tokens: 512 temperature: 1.0 top_p: 0.95 tags: - openai-compatible - multimodal - reasoning - tool-use base_image: # qwen38 includes unreleased gated-delta-network MTP fixes required by this recipe. image: "vllm/vllm-openai:qwen38@sha256:4a2f33a884222f7049b983263ad9976f89452bb81affecf5b67d89ad35c1bc31" weights: - source: "hf://Qwen/Qwen3.8-27B-FP8@017b9c7af6b5689d5dd426a76e0bc077eb5ca20a" mount_location: "/app/checkpoint/model" auth_secret_name: "hf_access_token" secrets: hf_access_token: null environment_variables: HF_HUB_ENABLE_HF_TRANSFER: "1" VLLM_LOGGING_LEVEL: WARNING VLLM_ENGINE_READY_TIMEOUT_S: "3600" docker_server: start_command: >- vllm serve /app/checkpoint/model --served-model-name Qwen/Qwen3.8-27B --host 0.0.0.0 --port 8000 --tensor-parallel-size 1 --max-model-len 262144 --kv-cache-dtype fp8 --gpu-memory-utilization 0.90 --max-num-seqs 64 --enable-prefix-caching --limit-mm-per-prompt.image 1 --mm-encoder-tp-mode data --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --speculative-config.method mtp --speculative-config.num_speculative_tokens 3 --kv-transfer-config.kv_connector SimpleCPUOffloadConnector --kv-transfer-config.kv_role kv_both --kv-transfer-config.kv_connector_extra_config.cpu_bytes_to_use_per_rank 68719476736 --kv-transfer-config.kv_connector_extra_config.lazy_offload false readiness_endpoint: /health liveness_endpoint: /health predict_endpoint: /v1/chat/completions server_port: 8000 resources: accelerator: H100:1 node_count: 1 cpu: "16" memory: 118Gi use_gpu: true runtime: predict_concurrency: 64 health_checks: restart_check_delay_seconds: 1800 restart_threshold_seconds: 1200 stop_traffic_threshold_seconds: 120 ``` This preset deploys the official FP8 checkpoint of Qwen3.8-27B through vLLM on a single H100. It enables MTP speculative decoding with three speculative tokens and CPU KV-cache offload, keeping time-to-first-token low while fitting the 262,144-token context window in a single GPU's memory. The deployment exposes an OpenAI-compatible chat completions endpoint with reasoning, tool calling, and one image per prompt. ## Flags The `start_command` passes these flags to the engine. Each one controls a runtime or serving behavior: | Flag | Value | What it does | | -------------------------------------------------------------------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `--tensor-parallel-size` | `1` | Number of GPUs to shard the model across. | | `--max-model-len` | `262144` | Maximum context length (tokens) the server accepts per request. | | `--kv-cache-dtype` | `fp8` | KV cache numeric precision. **fp8:** \~2× KV cache density with negligible quality impact on most models. | | `--gpu-memory-utilization` | `0.90` | Fraction of GPU memory vLLM may use for weights and KV cache. | | `--max-num-seqs` | `64` | Maximum number of concurrent sequences in the batch. | | `--enable-prefix-caching` | (no value) | Reuse KV cache across requests that share a prefix. | | `--limit-mm-per-prompt.image` | `1` | Maximum number of image inputs per prompt. | | `--mm-encoder-tp-mode` | `data` | Tensor-parallel mode for the multimodal vision encoder. **data:** Data-parallel encoding; each rank processes different images independently. | | `--reasoning-parser` | `qwen3` | Server-side parser that separates reasoning output into `reasoning_content`. **qwen3:** Qwen3-family thinking format (used by Qwen3, Qwen3.5, and Qwen3.6). | | `--enable-auto-tool-choice` | (no value) | Let the model choose when to call tools without requiring `tool_choice: "required"`. | | `--tool-call-parser` | `qwen3_coder` | Server-side parser that emits structured `tool_calls` on the response. **qwen3\_coder:** Qwen3-Coder tool format. | | `--speculative-config.method` | `mtp` | Speculative decoding method. **mtp:** Multi-token prediction head speculation. | | `--speculative-config.num_speculative_tokens` | `3` | Number of tokens the draft speculator proposes per step. | | `--kv-transfer-config.kv_connector` | `SimpleCPUOffloadConnector` | KV cache transfer connector for offloading cache to host memory or another node. **SimpleCPUOffloadConnector:** Offloads KV cache to CPU memory to extend the effective KV cache capacity beyond GPU memory. | | `--kv-transfer-config.kv_role` | `kv_both` | Role this node plays in KV cache transfer. **kv\_both:** Both sender and receiver, used for single-node CPU offload. | | `--kv-transfer-config.kv_connector_extra_config.cpu_bytes_to_use_per_rank` | `68719476736` | Bytes of CPU memory allocated per rank for KV cache offload. | | `--kv-transfer-config.kv_connector_extra_config.lazy_offload` | `false` | Whether to offload KV cache lazily (on demand) or eagerly. **false:** Eager offload; KV cache is copied to CPU memory as soon as it is produced. | ## Deploy Push the config to Baseten: ```sh theme={"system"} uvx truss push ``` You should see output similar to: ```output theme={"system"} ✨ Model qwen3.8-27b-latency was successfully pushed ✨ Model ID: abc1d2ef Deployment ID: xyz123 Endpoint: model-abc1d2ef.api.baseten.co Logs: https://app.baseten.co/models/abc1d2ef/logs/xyz123 ``` `truss push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable. ## Call the model Your deployment serves an OpenAI-compatible API. Now call your deployment to run inference: ```python main.py theme={"system"} import os from openai import OpenAI client = OpenAI( api_key=os.environ["BASETEN_API_KEY"], base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1", ) response = client.chat.completions.create( model="Qwen/Qwen3.8-27B", messages=[ {"role": "user", "content": "What is machine learning?"} ], ) print(response.choices[0].message.content) ``` ```sh theme={"system"} curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{ "model": "Qwen/Qwen3.8-27B", "messages": [ {"role": "user", "content": "What is machine learning?"} ] }' ``` To access the model's chain of thought, enable thinking mode. The server parses the reasoning output into a separate `reasoning_content` field on the response: ```python theme={"system"} response = client.chat.completions.create( model="Qwen/Qwen3.8-27B", messages=[ {"role": "user", "content": "How many r's in strawberry?"} ], extra_body={"chat_template_kwargs": {"enable_thinking": True}}, ) print(response.choices[0].message.reasoning_content) # chain of thought print(response.choices[0].message.content) # final answer ``` To let the model call tools, pass a `tools` array. The server returns structured `tool_calls` on the response: ```python theme={"system"} tools = [{ "type": "function", "function": { "name": "get_weather", "parameters": { "type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"], }, }, }] response = client.chat.completions.create( model="Qwen/Qwen3.8-27B", messages=[ {"role": "user", "content": "What's the weather in Paris?"} ], tools=tools, ) print(response.choices[0].message.tool_calls) ``` ## Next steps Endpoint anatomy, authentication, and sync versus async inference Scale replicas with traffic, including scale to zero