> ## Documentation Index
> Fetch the complete documentation index at: https://docs.baseten.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Qwen3.5

> Qwen3.5 recipes: 4 variants (4B, 9B, 35B, 122B), Dense, Hybrid MoE, and MoE architectures.

[Reasoning](https://docs.baseten.co/examples/models/capabilities/reasoning)
[Tool calling](https://docs.baseten.co/examples/models/capabilities/tool-calling)
[Long context](https://docs.baseten.co/examples/models/capabilities/long-context)
[Agentic](https://docs.baseten.co/examples/models/capabilities/agentic)

## Setup

Install the Baseten CLI and sign in, then install the OpenAI SDK.

**Install and sign in to Baseten**

**macOS or Linux**

```bash Terminal theme={"system"}
brew tap basetenlabs/baseten
brew trust --formula basetenlabs/baseten/baseten
brew install baseten
```

**Windows**

Download and extract the Windows archive for the latest Baseten CLI release:

```powershell Terminal theme={"system"}
$release = Invoke-RestMethod https://api.github.com/repos/basetenlabs/baseten-cli/releases/latest
$asset = $release.assets | Where-Object { $_.name -like "*_windows_amd64.zip" }
Invoke-WebRequest $asset.browser_download_url -OutFile baseten.zip
Expand-Archive -Force baseten.zip .
```

Move `baseten.exe` to a directory on your `PATH`.

For other platforms or a specific version, see the [Baseten CLI install reference](https://docs.baseten.co/reference/cli/baseten/overview.md#install).

```sh theme={"system"}
baseten auth login
```

**Install the OpenAI SDK**

```sh theme={"system"}
uv pip install openai
```

To install without Homebrew, see the [Baseten CLI install reference](https://docs.baseten.co/reference/cli/baseten/overview.md#install).

Pick the model you want to deploy. Each tab is a self-contained recipe.

**4B**

[Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) is a 4B-parameter dense model with up to 256K context.

This preset serves Qwen3.5-4B with BF16 weights on a single H100, optimized for low time-to-first-token.

> **Hardware**
>
> H100 × 1

> **Engine**
>
> vLLM (0.22.0-cu129 build)

> **Context**
>
> 32K

> **Concurrency**
>
> 128

## Write the config

Create and move into the project directory:

```sh theme={"system"}
mkdir qwen3.5-4b-latency && cd qwen3.5-4b-latency
```

Then create a file named `config.yaml` and paste the following:

```yaml config.yaml theme={"system"}
model_name: "model:qwen3.5-4b preset:latency"
model_metadata:
  description: >-
    Qwen 3.5 4B instruct (dense), OpenAI-compatible chat using vLLM with Qwen tooling parsers.
  repo_id: Qwen/Qwen3.5-4B
  example_model_input:
    model: "Qwen/Qwen3.5-4B"
    messages:
      - role: user
        content: "What is the capital of France?"
    stream: true
    max_tokens: 100
    temperature: 0.7
  tags:
    - openai-compatible
base_image:
  image: vllm/vllm-openai:v0.22.0-cu129
weights:
  - source: "hf://Qwen/Qwen3.5-4B@851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a"
    mount_location: "/app/checkpoint/model"
docker_server:
  start_command: >-
    sh -c 'GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) && vllm serve /app/checkpoint/model --tensor-parallel-size $GPU_COUNT
    --served-model-name Qwen/Qwen3.5-4B
    --host 0.0.0.0
    --port 8000
    --gpu-memory-utilization 0.95
    --max-model-len 32768
    --dtype bfloat16
    --reasoning-parser qwen3
    --enable-auto-tool-choice
    --tool-call-parser qwen3_coder
    --trust-remote-code
    --enable-prefix-caching
    --load-format runai_streamer'
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000
environment_variables:
  VLLM_LOGGING_LEVEL: WARNING
  VLLM_ENGINE_READY_TIMEOUT_S: "3600"
runtime:
  predict_concurrency: 128
  health_checks:
    startup_threshold_seconds: 1800
    restart_threshold_seconds: 1200
    stop_traffic_threshold_seconds: 120
resources:
  accelerator: H100:1
  use_gpu: true
```

## Flags

The `start_command` passes these flags to the engine. Each one controls a runtime or serving behavior:

| Flag | Value | What it does |
| - | - | - |
| `--tensor-parallel-size` | `$GPU_COUNT` | Number of GPUs to shard the model across. |
| `--gpu-memory-utilization` | `0.95` | Fraction of GPU memory vLLM may use for weights and KV cache. |
| `--max-model-len` | `32768` | Maximum context length (tokens) the server accepts per request. |
| `--dtype` | `bfloat16` | Weight precision loaded at runtime. **bfloat16:** BF16 weights, no quantization. |
| `--reasoning-parser` | `qwen3` | Server-side parser that separates reasoning output into `reasoning_content`. **qwen3:** Qwen3-family thinking format (used by Qwen3, Qwen3.5, and Qwen3.6). |
| `--enable-auto-tool-choice` | (no value) | Let the model choose when to call tools without requiring `tool_choice: "required"`. |
| `--tool-call-parser` | `qwen3_coder` | Server-side parser that emits structured `tool_calls` on the response. **qwen3\_coder:** Qwen3-Coder tool format. |
| `--trust-remote-code` | (no value) | Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures). |
| `--enable-prefix-caching` | (no value) | Reuse KV cache across requests that share a prefix. |
| `--load-format` | `runai_streamer` | Weight loading backend. **runai\_streamer:** Stream weights from object storage without materializing to disk. |

## Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

```sh Baseten CLI theme={"system"}
baseten model push
```

```sh Truss CLI theme={"system"}
uvx truss push
```

You should see output similar to:

```output theme={"system"}
Pushing model "qwen3.5-4b-latency"...
Uploading model...
Uploaded model in 0s
✨ Model qwen3.5-4b-latency was successfully pushed ✨

  Model:       qwen3.5-4b-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef
```

`baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable.

## Call the model

Your deployment serves an OpenAI-compatible API.

Now call your deployment to run inference:

**Python**

```python main.py theme={"system"}
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.5-4B",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)
```

**cURL**

```sh theme={"system"}
curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "Qwen/Qwen3.5-4B",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'
```

To access the model's chain of thought, enable thinking mode. The server parses the reasoning output into a separate `reasoning_content` field on the response:

```python theme={"system"}
response = client.chat.completions.create(
    model="Qwen/Qwen3.5-4B",
    messages=[
        {"role": "user", "content": "How many r's in strawberry?"}
    ],
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
print(response.choices[0].message.reasoning_content)  # chain of thought
print(response.choices[0].message.content)            # final answer
```

To let the model call tools, pass a `tools` array. The server returns structured `tool_calls` on the response:

```python theme={"system"}
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "parameters": {
            "type": "object",
            "properties": {"location": {"type": "string"}},
            "required": ["location"],
        },
    },
}]

response = client.chat.completions.create(
    model="Qwen/Qwen3.5-4B",
    messages=[
        {"role": "user", "content": "What's the weather in Paris?"}
    ],
    tools=tools,
)
print(response.choices[0].message.tool_calls)
```

**9B**

[Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) is a 9B-parameter dense model with up to 256K context.

This preset serves Qwen3.5-9B with BF16 weights on a single H100. It's the smallest dense Qwen3.5 deployment that keeps reasoning and tool calling enabled.

> **Hardware**
>
> H100 × 1

> **Engine**
>
> vLLM (0.22.0-cu129 build)

> **Context**
>
> 32K

> **Concurrency**
>
> 128

## Write the config

Create and move into the project directory:

```sh theme={"system"}
mkdir qwen3.5-9b-latency && cd qwen3.5-9b-latency
```

Then create a file named `config.yaml` and paste the following:

```yaml config.yaml theme={"system"}
model_name: "model:qwen3.5-9b preset:latency"
model_metadata:
  description: >-
    Qwen 3.5 9B instruct (dense), OpenAI-compatible chat using vLLM with Qwen tooling parsers.
  repo_id: Qwen/Qwen3.5-9B
  example_model_input:
    model: "Qwen/Qwen3.5-9B"
    messages:
      - role: user
        content: "What is the capital of France?"
    stream: true
    max_tokens: 100
    temperature: 0.7
  tags:
    - openai-compatible
base_image:
  image: vllm/vllm-openai:v0.22.0-cu129
weights:
  - source: "hf://Qwen/Qwen3.5-9B@c202236235762e1c871ad0ccb60c8ee5ba337b9a"
    mount_location: "/app/checkpoint/model"
docker_server:
  start_command: >-
    sh -c 'GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) && vllm serve /app/checkpoint/model --tensor-parallel-size $GPU_COUNT
    --served-model-name Qwen/Qwen3.5-9B
    --host 0.0.0.0
    --port 8000
    --gpu-memory-utilization 0.95
    --max-model-len 32768
    --dtype bfloat16
    --reasoning-parser qwen3
    --enable-auto-tool-choice
    --tool-call-parser qwen3_coder
    --trust-remote-code
    --enable-prefix-caching
    --load-format runai_streamer'
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000
environment_variables:
  VLLM_LOGGING_LEVEL: WARNING
  VLLM_ENGINE_READY_TIMEOUT_S: "3600"
runtime:
  predict_concurrency: 128
  health_checks:
    startup_threshold_seconds: 1800
    restart_threshold_seconds: 1200
    stop_traffic_threshold_seconds: 120
resources:
  accelerator: H100:1
  use_gpu: true
```

## Flags

The `start_command` passes these flags to the engine. Each one controls a runtime or serving behavior:

| Flag | Value | What it does |
| - | - | - |
| `--tensor-parallel-size` | `$GPU_COUNT` | Number of GPUs to shard the model across. |
| `--gpu-memory-utilization` | `0.95` | Fraction of GPU memory vLLM may use for weights and KV cache. |
| `--max-model-len` | `32768` | Maximum context length (tokens) the server accepts per request. |
| `--dtype` | `bfloat16` | Weight precision loaded at runtime. **bfloat16:** BF16 weights, no quantization. |
| `--reasoning-parser` | `qwen3` | Server-side parser that separates reasoning output into `reasoning_content`. **qwen3:** Qwen3-family thinking format (used by Qwen3, Qwen3.5, and Qwen3.6). |
| `--enable-auto-tool-choice` | (no value) | Let the model choose when to call tools without requiring `tool_choice: "required"`. |
| `--tool-call-parser` | `qwen3_coder` | Server-side parser that emits structured `tool_calls` on the response. **qwen3\_coder:** Qwen3-Coder tool format. |
| `--trust-remote-code` | (no value) | Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures). |
| `--enable-prefix-caching` | (no value) | Reuse KV cache across requests that share a prefix. |
| `--load-format` | `runai_streamer` | Weight loading backend. **runai\_streamer:** Stream weights from object storage without materializing to disk. |

## Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

```sh Baseten CLI theme={"system"}
baseten model push
```

```sh Truss CLI theme={"system"}
uvx truss push
```

You should see output similar to:

```output theme={"system"}
Pushing model "qwen3.5-9b-latency"...
Uploading model...
Uploaded model in 0s
✨ Model qwen3.5-9b-latency was successfully pushed ✨

  Model:       qwen3.5-9b-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef
```

`baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable.

## Call the model

Your deployment serves an OpenAI-compatible API.

Now call your deployment to run inference:

**Python**

```python main.py theme={"system"}
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.5-9B",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)
```

**cURL**

```sh theme={"system"}
curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "Qwen/Qwen3.5-9B",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'
```

To access the model's chain of thought, enable thinking mode. The server parses the reasoning output into a separate `reasoning_content` field on the response:

```python theme={"system"}
response = client.chat.completions.create(
    model="Qwen/Qwen3.5-9B",
    messages=[
        {"role": "user", "content": "How many r's in strawberry?"}
    ],
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
print(response.choices[0].message.reasoning_content)  # chain of thought
print(response.choices[0].message.content)            # final answer
```

To let the model call tools, pass a `tools` array. The server returns structured `tool_calls` on the response:

```python theme={"system"}
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "parameters": {
            "type": "object",
            "properties": {"location": {"type": "string"}},
            "required": ["location"],
        },
    },
}]

response = client.chat.completions.create(
    model="Qwen/Qwen3.5-9B",
    messages=[
        {"role": "user", "content": "What's the weather in Paris?"}
    ],
    tools=tools,
)
print(response.choices[0].message.tool_calls)
```

**35B**

[Qwen/Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B) is a 35B-parameter hybrid MoE model (3B active per token) with up to 256K context.

This variant ships in 2 presets tuned for different goals: **Throughput** for highest tokens per second, and **Latency** for lowest time-to-first-token. Pick the tab that matches your workload.

**Throughput**

This preset serves Qwen3.5-35B FP8 on a single B200, with prefix caching and chunked prefill enabled. It maximizes aggregate throughput at high concurrency with minor quality impact from FP8.

> **Hardware**
>
> B200

> **Engine**
>
> vLLM 0.22.0

> **Context**
>
> 256K

> **Concurrency**
>
> 1000

## Write the config

Create and move into the project directory:

```sh theme={"system"}
mkdir qwen3.5-35b-throughput && cd qwen3.5-35b-throughput
```

Then create a file named `config.yaml` and paste the following:

```yaml config.yaml theme={"system"}
########################################################
# Remove ( --language-model-only ) from the start command to turn on multimodal mode
########################################################
model_name: "model:qwen3.5-35b preset:throughput"
model_metadata:
  description: >-
    Qwen 3.5 35B A3B FP8 MoE throughput on B200, language-only mode (--language-model-only) optional.
  repo_id: Qwen/Qwen3.5-35B-A3B-FP8
  example_model_input:
    model: "Qwen/Qwen3.5-35B-A3B-FP8"
    messages:
      - role: user
        content: "What is the capital of France?"
    stream: true
    max_tokens: 100
    temperature: 0.7
  tags:
    - openai-compatible
base_image:
  image: vllm/vllm-openai:v0.22.0
weights:
  - source: "hf://Qwen/Qwen3.5-35B-A3B-FP8@9d1823d2dee688a6b25e77009dc727688c44936e"
    mount_location: "/app/checkpoint/model"
environment_variables:
  VLLM_LOGGING_LEVEL: WARNING
  VLLM_ENGINE_READY_TIMEOUT_S: "3600"
  VLLM_USE_FLASHINFER_MOE_FP8: "0"
  PYTORCH_ALLOC_CONF: "expandable_segments:True"
docker_server:
  start_command: >-
    sh -c 'GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) && vllm serve /app/checkpoint/model
    --tensor-parallel-size $GPU_COUNT
    --served-model-name Qwen/Qwen3.5-35B-A3B-FP8
    --host 0.0.0.0
    --port 8000
    --language-model-only
    --gpu-memory-utilization 0.95
    --kv-cache-dtype fp8
    --reasoning-parser qwen3
    --enable-chunked-prefill
    --enable-prefix-caching
    --max-num-seqs 512
    --trust-remote-code
    --load-format runai_streamer'
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000
runtime:
  predict_concurrency: 1000
  health_checks:
    startup_threshold_seconds: 1800
    restart_threshold_seconds: 1200
    stop_traffic_threshold_seconds: 120
resources:
  accelerator: B200
  use_gpu: true
```

## Flags

The `start_command` passes these flags to the engine. Each one controls a runtime or serving behavior:

| Flag | Value | What it does |
| - | - | - |
| `--tensor-parallel-size` | `$GPU_COUNT` | Number of GPUs to shard the model across. |
| `--language-model-only` | (no value) | Disable the multimodal path; text-only serving. Remove to enable image/video inputs. |
| `--gpu-memory-utilization` | `0.95` | Fraction of GPU memory vLLM may use for weights and KV cache. |
| `--kv-cache-dtype` | `fp8` | KV cache numeric precision. **fp8:** \~2× KV cache density with negligible quality impact on most models. |
| `--reasoning-parser` | `qwen3` | Server-side parser that separates reasoning output into `reasoning_content`. **qwen3:** Qwen3-family thinking format (used by Qwen3, Qwen3.5, and Qwen3.6). |
| `--enable-chunked-prefill` | (no value) | Process long prompts in chunks so decode requests keep running. |
| `--enable-prefix-caching` | (no value) | Reuse KV cache across requests that share a prefix. |
| `--max-num-seqs` | `512` | Maximum number of concurrent sequences in the batch. |
| `--trust-remote-code` | (no value) | Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures). |
| `--load-format` | `runai_streamer` | Weight loading backend. **runai\_streamer:** Stream weights from object storage without materializing to disk. |

## Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

```sh Baseten CLI theme={"system"}
baseten model push
```

```sh Truss CLI theme={"system"}
uvx truss push
```

You should see output similar to:

```output theme={"system"}
Pushing model "qwen3.5-35b-throughput"...
Uploading model...
Uploaded model in 0s
✨ Model qwen3.5-35b-throughput was successfully pushed ✨

  Model:       qwen3.5-35b-throughput (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef
```

`baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable.

## Call the model

Your deployment serves an OpenAI-compatible API.

Now call your deployment to run inference:

**Python**

```python main.py theme={"system"}
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.5-35B-A3B-FP8",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)
```

**cURL**

```sh theme={"system"}
curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "Qwen/Qwen3.5-35B-A3B-FP8",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'
```

To access the model's chain of thought, enable thinking mode. The server parses the reasoning output into a separate `reasoning_content` field on the response:

```python theme={"system"}
response = client.chat.completions.create(
    model="Qwen/Qwen3.5-35B-A3B-FP8",
    messages=[
        {"role": "user", "content": "How many r's in strawberry?"}
    ],
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
print(response.choices[0].message.reasoning_content)  # chain of thought
print(response.choices[0].message.content)            # final answer
```

**Latency**

This preset serves Qwen3.5-35B with FP8 weights on H100:2, optimized for low time-to-first-token on interactive chat and short-horizon agent workflows.

> **Hardware**
>
> H100 × 2

> **Engine**
>
> vLLM (0.22.0-cu129 build)

> **Context**
>
> 32K

> **Concurrency**
>
> 128

## Write the config

Create and move into the project directory:

```sh theme={"system"}
mkdir qwen3.5-35b-latency && cd qwen3.5-35b-latency
```

Then create a file named `config.yaml` and paste the following:

```yaml config.yaml theme={"system"}
model_name: "model:qwen3.5-35b preset:latency"
model_metadata:
  description: >-
    Qwen 3.5 35B A3B MoE instruct FP8 weights, TP=2 latency preset with Qwen parsers.
  repo_id: Qwen/Qwen3.5-35B-A3B-FP8
  example_model_input:
    model: "Qwen/Qwen3.5-35B-A3B-FP8"
    messages:
      - role: user
        content: "What is the capital of France?"
    stream: true
    max_tokens: 100
    temperature: 0.7
  tags:
    - openai-compatible
base_image:
  image: vllm/vllm-openai:v0.22.0-cu129
weights:
  - source: "hf://Qwen/Qwen3.5-35B-A3B-FP8@main"
    mount_location: "/app/checkpoint/model"
    auth_secret_name: "hf_access_token"
secrets:
  hf_access_token: null
docker_server:
  start_command: >-
    sh -c 'GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) && vllm serve /app/checkpoint/model
    --served-model-name Qwen/Qwen3.5-35B-A3B-FP8
    --host 0.0.0.0
    --port 8000
    --gpu-memory-utilization 0.95
    --max-model-len 32768
    --kv-cache-dtype fp8
    --tensor-parallel-size $GPU_COUNT
    --reasoning-parser qwen3
    --enable-auto-tool-choice
    --tool-call-parser qwen3_coder
    --trust-remote-code
    --enable-prefix-caching
    --load-format runai_streamer'
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000
environment_variables:
  VLLM_LOGGING_LEVEL: WARNING
  VLLM_ENGINE_READY_TIMEOUT_S: "3600"
runtime:
  predict_concurrency: 128
  health_checks:
    startup_threshold_seconds: 1800
    restart_threshold_seconds: 1200
    stop_traffic_threshold_seconds: 120
resources:
  accelerator: H100:2
  use_gpu: true
```

## Flags

The `start_command` passes these flags to the engine. Each one controls a runtime or serving behavior:

| Flag | Value | What it does |
| - | - | - |
| `--gpu-memory-utilization` | `0.95` | Fraction of GPU memory vLLM may use for weights and KV cache. |
| `--max-model-len` | `32768` | Maximum context length (tokens) the server accepts per request. |
| `--kv-cache-dtype` | `fp8` | KV cache numeric precision. **fp8:** \~2× KV cache density with negligible quality impact on most models. |
| `--tensor-parallel-size` | `$GPU_COUNT` | Number of GPUs to shard the model across. |
| `--reasoning-parser` | `qwen3` | Server-side parser that separates reasoning output into `reasoning_content`. **qwen3:** Qwen3-family thinking format (used by Qwen3, Qwen3.5, and Qwen3.6). |
| `--enable-auto-tool-choice` | (no value) | Let the model choose when to call tools without requiring `tool_choice: "required"`. |
| `--tool-call-parser` | `qwen3_coder` | Server-side parser that emits structured `tool_calls` on the response. **qwen3\_coder:** Qwen3-Coder tool format. |
| `--trust-remote-code` | (no value) | Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures). |
| `--enable-prefix-caching` | (no value) | Reuse KV cache across requests that share a prefix. |
| `--load-format` | `runai_streamer` | Weight loading backend. **runai\_streamer:** Stream weights from object storage without materializing to disk. |

## Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

```sh Baseten CLI theme={"system"}
baseten model push
```

```sh Truss CLI theme={"system"}
uvx truss push
```

You should see output similar to:

```output theme={"system"}
Pushing model "qwen3.5-35b-latency"...
Uploading model...
Uploaded model in 0s
✨ Model qwen3.5-35b-latency was successfully pushed ✨

  Model:       qwen3.5-35b-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef
```

`baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable.

## Call the model

Your deployment serves an OpenAI-compatible API.

Now call your deployment to run inference:

**Python**

```python main.py theme={"system"}
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.5-35B-A3B-FP8",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)
```

**cURL**

```sh theme={"system"}
curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "Qwen/Qwen3.5-35B-A3B-FP8",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'
```

To access the model's chain of thought, enable thinking mode. The server parses the reasoning output into a separate `reasoning_content` field on the response:

```python theme={"system"}
response = client.chat.completions.create(
    model="Qwen/Qwen3.5-35B-A3B-FP8",
    messages=[
        {"role": "user", "content": "How many r's in strawberry?"}
    ],
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
print(response.choices[0].message.reasoning_content)  # chain of thought
print(response.choices[0].message.content)            # final answer
```

To let the model call tools, pass a `tools` array. The server returns structured `tool_calls` on the response:

```python theme={"system"}
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "parameters": {
            "type": "object",
            "properties": {"location": {"type": "string"}},
            "required": ["location"],
        },
    },
}]

response = client.chat.completions.create(
    model="Qwen/Qwen3.5-35B-A3B-FP8",
    messages=[
        {"role": "user", "content": "What's the weather in Paris?"}
    ],
    tools=tools,
)
print(response.choices[0].message.tool_calls)
```

**122B**

[Qwen/Qwen3.5-122B-A10B](https://huggingface.co/Qwen/Qwen3.5-122B-A10B) is a 122B-parameter MoE model (10B active per token) with up to 256K context.

This preset serves Qwen3.5-122B with FP8 weights on H100:4. It keeps time-to-first-token low while fitting the full model on a single H100 node.

> **Hardware**
>
> H100 × 4

> **Engine**
>
> vLLM (0.22.0-cu129 build)

> **Context**
>
> 32K

> **Concurrency**
>
> 128

## Write the config

Create and move into the project directory:

```sh theme={"system"}
mkdir qwen3.5-122b-latency && cd qwen3.5-122b-latency
```

Then create a file named `config.yaml` and paste the following:

```yaml config.yaml theme={"system"}
model_name: "model:qwen3.5-122b preset:latency"
model_metadata:
  description: >-
    Qwen 3.5 122B A10B MoE instruct FP8 weights, TP=4 latency preset using vLLM with Qwen parsers.
  repo_id: Qwen/Qwen3.5-122B-A10B-FP8
  example_model_input:
    model: "Qwen/Qwen3.5-122B-A10B-FP8"
    messages:
      - role: user
        content: "What is the capital of France?"
    stream: true
    max_tokens: 100
    temperature: 0.7
  tags:
    - openai-compatible
base_image:
  image: vllm/vllm-openai:v0.22.0-cu129
weights:
  - source: "hf://Qwen/Qwen3.5-122B-A10B-FP8@main"
    mount_location: "/app/checkpoint/model"
docker_server:
  start_command: >-
    sh -c 'GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) && vllm serve /app/checkpoint/model
    --served-model-name Qwen/Qwen3.5-122B-A10B-FP8
    --host 0.0.0.0
    --port 8000
    --gpu-memory-utilization 0.95
    --max-model-len 32768
    --kv-cache-dtype fp8
    --tensor-parallel-size $GPU_COUNT
    --reasoning-parser qwen3
    --enable-auto-tool-choice
    --tool-call-parser qwen3_coder
    --trust-remote-code
    --enable-prefix-caching
    --load-format runai_streamer'
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000
environment_variables:
  VLLM_LOGGING_LEVEL: WARNING
  VLLM_ENGINE_READY_TIMEOUT_S: "3600"
runtime:
  predict_concurrency: 128
  health_checks:
    startup_threshold_seconds: 1800
    restart_threshold_seconds: 1200
    stop_traffic_threshold_seconds: 120
resources:
  accelerator: H100:4
  use_gpu: true
```

## Flags

The `start_command` passes these flags to the engine. Each one controls a runtime or serving behavior:

| Flag | Value | What it does |
| - | - | - |
| `--gpu-memory-utilization` | `0.95` | Fraction of GPU memory vLLM may use for weights and KV cache. |
| `--max-model-len` | `32768` | Maximum context length (tokens) the server accepts per request. |
| `--kv-cache-dtype` | `fp8` | KV cache numeric precision. **fp8:** \~2× KV cache density with negligible quality impact on most models. |
| `--tensor-parallel-size` | `$GPU_COUNT` | Number of GPUs to shard the model across. |
| `--reasoning-parser` | `qwen3` | Server-side parser that separates reasoning output into `reasoning_content`. **qwen3:** Qwen3-family thinking format (used by Qwen3, Qwen3.5, and Qwen3.6). |
| `--enable-auto-tool-choice` | (no value) | Let the model choose when to call tools without requiring `tool_choice: "required"`. |
| `--tool-call-parser` | `qwen3_coder` | Server-side parser that emits structured `tool_calls` on the response. **qwen3\_coder:** Qwen3-Coder tool format. |
| `--trust-remote-code` | (no value) | Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures). |
| `--enable-prefix-caching` | (no value) | Reuse KV cache across requests that share a prefix. |
| `--load-format` | `runai_streamer` | Weight loading backend. **runai\_streamer:** Stream weights from object storage without materializing to disk. |

## Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

```sh Baseten CLI theme={"system"}
baseten model push
```

```sh Truss CLI theme={"system"}
uvx truss push
```

You should see output similar to:

```output theme={"system"}
Pushing model "qwen3.5-122b-latency"...
Uploading model...
Uploaded model in 0s
✨ Model qwen3.5-122b-latency was successfully pushed ✨

  Model:       qwen3.5-122b-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef
```

`baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable.

## Call the model

Your deployment serves an OpenAI-compatible API.

Now call your deployment to run inference:

**Python**

```python main.py theme={"system"}
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.5-122B-A10B-FP8",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)
```

**cURL**

```sh theme={"system"}
curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "Qwen/Qwen3.5-122B-A10B-FP8",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'
```

To access the model's chain of thought, enable thinking mode. The server parses the reasoning output into a separate `reasoning_content` field on the response:

```python theme={"system"}
response = client.chat.completions.create(
    model="Qwen/Qwen3.5-122B-A10B-FP8",
    messages=[
        {"role": "user", "content": "How many r's in strawberry?"}
    ],
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
print(response.choices[0].message.reasoning_content)  # chain of thought
print(response.choices[0].message.content)            # final answer
```

To let the model call tools, pass a `tools` array. The server returns structured `tool_calls` on the response:

```python theme={"system"}
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "parameters": {
            "type": "object",
            "properties": {"location": {"type": "string"}},
            "required": ["location"],
        },
    },
}]

response = client.chat.completions.create(
    model="Qwen/Qwen3.5-122B-A10B-FP8",
    messages=[
        {"role": "user", "content": "What's the weather in Paris?"}
    ],
    tools=tools,
)
print(response.choices[0].message.tool_calls)
```

## Next steps

> **[Call your model](https://docs.baseten.co/inference/calling-your-model.md)**
>
> Endpoint anatomy, authentication, and sync versus async inference

> **[Autoscaling](https://docs.baseten.co/deployment/autoscaling/overview.md)**
>
> Scale replicas with traffic, including scale to zero