> ## Documentation Index
> Fetch the complete documentation index at: https://docs.baseten.co/llms.txt
> Use this file to discover all available pages before exploring further.
# GLM-5
> Z.ai's GLM-5 reasoning model, served from an FP8 checkpoint on B200:8.
Z.ai's GLM-5 reasoning model, served from an FP8 checkpoint on B200:8.
## Setup
Install the Baseten CLI and sign in, then install the OpenAI SDK.
**Install and sign in to Baseten**
```bash Terminal theme={"system"}
brew tap basetenlabs/baseten
brew trust --formula basetenlabs/baseten/baseten
brew install baseten
```
Download and extract the Windows archive for the latest Baseten CLI release:
```powershell Terminal theme={"system"}
$release = Invoke-RestMethod https://api.github.com/repos/basetenlabs/baseten-cli/releases/latest
$asset = $release.assets | Where-Object { $_.name -like "*_windows_amd64.zip" }
Invoke-WebRequest $asset.browser_download_url -OutFile baseten.zip
Expand-Archive -Force baseten.zip .
```
Move `baseten.exe` to a directory on your `PATH`.
For other platforms or a specific version, see the [Baseten CLI install reference](/reference/cli/baseten/overview#install).
```sh theme={"system"}
baseten auth login
```
**Install the OpenAI SDK**
```sh theme={"system"}
uv pip install openai
```
To install without Homebrew, see the [Baseten CLI install reference](/reference/cli/baseten/overview#install).
This preset serves GLM-5 FP8 on B200:8, Z.ai's frontier model tuned for the lowest time-to-first-token available.
B200 × 8
vLLM 0.22.0
128K
64
## Write the config
Create and move into the project directory:
```sh theme={"system"}
mkdir glm-5-latency && cd glm-5-latency
```
Then create a file named `config.yaml` and paste the following:
```yaml config.yaml theme={"system"}
model_metadata:
example_model_input:
messages:
- role: system
content: "You are a helpful assistant."
- role: user
content: "What is the meaning of life?"
stream: true
model: zai-org/GLM-5
max_tokens: 32768
temperature: 0.7
tags:
- openai-compatible
model_name: "model:glm-5 preset:latency"
base_image:
image: vllm/vllm-openai:v0.22.0
docker_server:
start_command: >-
sh -c "VLLM_DEEP_GEMM_WARMUP=relax python3 -m vllm.entrypoints.openai.api_server
--model /models/GLM-5-FP8
--chat-template /models/GLM-5-FP8/chat_template.jinja
--host 0.0.0.0 --port 8000
--served-model-name zai-org/GLM-5
--tensor-parallel-size 8
--trust-remote-code
--load-format runai_streamer
--disable-log-stats
--max-num-seqs 64
--max-num-batched-tokens 8192
--tool-call-parser glm47
--reasoning-parser glm45
--enable-auto-tool-choice
--speculative-config.method mtp
--speculative-config.num_speculative_tokens 1"
readiness_endpoint: /health
liveness_endpoint: /health
predict_endpoint: /v1/chat/completions
server_port: 8000
weights:
- source: "hf://zai-org/GLM-5-FP8@4f96cc5eec29dcee5d6ded54f7ffe889438f9516"
mount_location: "/models/GLM-5-FP8"
ignore_patterns:
- "*.md"
- "*.txt"
resources:
accelerator: B200:8
use_gpu: true
runtime:
predict_concurrency: 64
health_checks:
startup_threshold_seconds: 1800
restart_threshold_seconds: 1200
stop_traffic_threshold_seconds: 120
```
## Flags
The `start_command` passes these flags to the engine. Each one controls a runtime or serving behavior:
| Flag | Value | What it does |
| - | - | - |
| `--model` | `/models/GLM-5-FP8` | Path (or HF repo) the engine loads the model from. |
| `--chat-template` | `/models/GLM-5-FP8/chat_template.jinja` | Path to a Jinja chat template file that overrides the checkpoint's default. |
| `--tensor-parallel-size` | `8` | Number of GPUs to shard the model across. |
| `--trust-remote-code` | (no value) | Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures). |
| `--load-format` | `runai_streamer` | Weight loading backend. **runai\_streamer:** Stream weights from object storage without materializing to disk. |
| `--disable-log-stats` | (no value) | Suppress periodic engine stats logging. |
| `--max-num-seqs` | `64` | Maximum number of concurrent sequences in the batch. |
| `--max-num-batched-tokens` | `8192` | Maximum total tokens processed per scheduler step. |
| `--tool-call-parser` | `glm47` | Server-side parser that emits structured `tool_calls` on the response. **glm47:** GLM-4.7-style tool format, also used by GLM-5 and GLM-5.3. |
| `--reasoning-parser` | `glm45` | Server-side parser that separates reasoning output into `reasoning_content`. **glm45:** GLM-4.5-style thinking format, also used by GLM-5 and GLM-5.3. |
| `--enable-auto-tool-choice` | (no value) | Let the model choose when to call tools without requiring `tool_choice: "required"`. |
| `--speculative-config.method` | `mtp` | Speculative decoding method. **mtp:** Multi-token prediction head speculation. |
| `--speculative-config.num_speculative_tokens` | `1` | Number of tokens the draft speculator proposes per step. |
## Deploy
Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:
```sh Baseten CLI theme={"system"}
baseten model push
```
```sh Truss CLI theme={"system"}
uvx truss push
```
You should see output similar to:
```output theme={"system"}
Pushing model "glm-5-latency"...
Uploading model...
Uploaded model in 0s
✨ Model glm-5-latency was successfully pushed ✨
Model: glm-5-latency (abc1d2ef)
Deployment: xyz123
Environment: production
🪵 View logs:
deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed)
app: https://app.baseten.co/models/abc1d2ef/logs/xyz123
🚀 Invoke your model:
URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
CLI: baseten model predict --model-id abc1d2ef
```
`baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable.
## Call the model
Your deployment serves an OpenAI-compatible API.
Now call your deployment to run inference:
```python main.py theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["BASETEN_API_KEY"],
base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)
response = client.chat.completions.create(
model="zai-org/GLM-5",
messages=[
{"role": "user", "content": "What is machine learning?"}
],
)
print(response.choices[0].message.content)
```
```sh theme={"system"}
curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $BASETEN_API_KEY" \
-d '{
"model": "zai-org/GLM-5",
"messages": [
{"role": "user", "content": "What is machine learning?"}
]
}'
```
The server parses the model's chain of thought into a separate `reasoning_content` field on the response. Read it alongside the final answer:
```python theme={"system"}
response = client.chat.completions.create(
model="zai-org/GLM-5",
messages=[
{"role": "user", "content": "How many r's in strawberry?"}
],
)
print(response.choices[0].message.reasoning_content) # chain of thought
print(response.choices[0].message.content) # final answer
```
To let the model call tools, pass a `tools` array. The server returns structured `tool_calls` on the response:
```python theme={"system"}
tools = [{
"type": "function",
"function": {
"name": "get_weather",
"parameters": {
"type": "object",
"properties": {"location": {"type": "string"}},
"required": ["location"],
},
},
}]
response = client.chat.completions.create(
model="zai-org/GLM-5",
messages=[
{"role": "user", "content": "What's the weather in Paris?"}
],
tools=tools,
)
print(response.choices[0].message.tool_calls)
```
## Next steps
Endpoint anatomy, authentication, and sync versus async inference
Scale replicas with traffic, including scale to zero