> ## Documentation Index
> Fetch the complete documentation index at: https://docs.baseten.co/llms.txt
> Use this file to discover all available pages before exploring further.
# GLM-4.7
> GLM-4.7 recipes: 2 variants (Standard, Flash), MoE architecture.
## Setup
To get started, sign into Baseten with Truss and then install the OpenAI SDK.
**Sign in to Baseten**
```sh theme={"system"}
uvx truss login --browser
```
**Install the OpenAI SDK**
```sh theme={"system"}
uv pip install openai
```
Pick the model you want to deploy. Each tab is a self-contained recipe.
[zai-org/GLM-4.7](https://huggingface.co/zai-org/GLM-4.7) is a MoE model with up to 198K context.
This preset serves GLM-4.7 from an FP4 checkpoint on B200:4, delivering frontier-class reasoning at single-node cost.
B200 × 4
TRT-LLM v2
198K
64
## Write the config
Create and move into the project directory:
```sh theme={"system"}
mkdir glm-4.7-latency && cd glm-4.7-latency
```
Then create a file named `config.yaml` and paste the following:
```yaml config.yaml theme={"system"}
model_name: "model:glm-4.7 preset:latency"
resources:
accelerator: B200:4
cpu: "1"
memory: 10Gi
use_gpu: true
model_metadata:
example_model_input:
{
"model": "glm47",
"messages":
[
{
"role": "user",
"content": "Given an array of integers nums and an integer target, return indices of the two numbers such that they add up to target. You may assume that each input would have exactly one solution, and you may not use the same element twice. You can return the answer in any order. class Solution: def twoSum(self, nums: List[int], target: int) -> List[int]:",
},
],
"stream": true,
"max_tokens": 2048,
"temperature": 0.5,
}
weights:
- source: "hf://baseten-admin/glm-4.7-fp4@main"
mount_location: "/app/model_cache/glm47"
auth_secret_name: "hf_access_token"
trt_llm:
build:
checkpoint_repository:
# repo: baseten-admin/glm-4.7-fp4
repo: michaelfeil/empty-model
revision: main
source: HF
inference_stack: v2
runtime:
enable_chunked_prefill: true
max_batch_size: 64
max_num_tokens: 8192
max_seq_len: 202752
tensor_parallel_size: 4
served_model_name: glm47
patch_kwargs:
disable_overlap_scheduler: True
guided_decoding_backend: "xgrammar"
moe_expert_parallel_size: 4
moe_config:
use_low_precision_moe_combine: true
backend: TRTLLM
kv_cache_config:
free_gpu_memory_fraction: 0.8
enable_block_reuse: true
# host_cache_size: 100000000000
cuda_graph_config:
enable_padding: false
enable_iter_perf_stats: true
autotuner_enabled: false
model_path: /app/model_cache/glm47
```
## Key parameters
[Baseten Inference Stack](/engines/bis-llm/overview) (BIS) reads these fields from the `trt_llm` block. Each one shapes how the engine is built and served:
| Parameter | Value |
| -------------------- | --------- |
| Tensor parallel size | `4` |
| Max sequence length | `202752` |
| Max batch size | `64` |
| Max batched tokens | `8192` |
| Chunked prefill | `enabled` |
| Inference stack | `v2` |
| Served model name | `glm47` |
## Deploy
Push the config to Baseten:
```sh theme={"system"}
uvx truss push
```
You should see output similar to:
```text theme={"system"}
✨ Model glm-4.7-latency was successfully pushed ✨
🪵 View logs for your deployment at https://app.baseten.co/models/abcd1234/logs/wxyz5678
```
Your **model ID** is the string after `/models/` in the logs URL (`abcd1234` in the example). Use it wherever you see `{model_id}` in the next section.
## Call the model
Your deployment serves an OpenAI-compatible API. Replace `{model_id}` with your model ID and make sure `BASETEN_API_KEY` is set.
Now call your deployment to run inference:
```python main.py theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["BASETEN_API_KEY"],
base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)
response = client.chat.completions.create(
model="glm47",
messages=[
{"role": "user", "content": "What is machine learning?"}
],
)
print(response.choices[0].message.content)
```
```sh theme={"system"}
curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $BASETEN_API_KEY" \
-d '{
"model": "glm47",
"messages": [
{"role": "user", "content": "What is machine learning?"}
]
}'
```
[zai-org/GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash) is a MoE model with up to 128K context.
This preset serves GLM-4.7 Flash on H100:2 with the glm47 tool-call parser enabled, tuned for latency-sensitive agent workflows.
H100 × 2
vLLM (latest build)
128K
32
## Write the config
Create and move into the project directory:
```sh theme={"system"}
mkdir glm-4.7-flash-latency && cd glm-4.7-flash-latency
```
Then create a file named `config.yaml` and paste the following:
```yaml config.yaml theme={"system"}
base_image:
image: vllm/vllm-openai:latest
build_commands:
- pip install -U --pre transformers==5.0.0
docker_server:
liveness_endpoint: /health
predict_endpoint: /v1/chat/completions
readiness_endpoint: /health
server_port: 8000
start_command: vllm serve zai-org/GLM-4.7-Flash --tensor-parallel-size 2 --gpu-memory-utilization 0.8 --tool-call-parser glm47 --enable-auto-tool-choice --served-model-name zai-org/GLM-4.7-Flash --host 0.0.0.0 --port 8000 --trust-remote-code --max-model-len auto
model_metadata:
example_model_input:
max_tokens: 32768
messages:
- content: You are a helpful assistant.
role: system
- content: What is the meaning of life?
role: user
model: zai-org/GLM-4.7-Flash
stream: true
temperature: 0.7
tags:
- openai-compatible
model_name: "model:glm-4.7-flash preset:latency"
python_version: py39
resources:
accelerator: H100:2
cpu: '1'
memory: 2Gi
use_gpu: true
runtime:
is_websocket_endpoint: false
predict_concurrency: 32
transport:
kind: http
```
## Flags
The `start_command` passes these flags to the engine. Each one controls a runtime or serving behavior:
| Flag | Value | What it does |
| --------------------------- | ---------- | ---------------------------------------------------------------------------------------------------------- |
| `--tensor-parallel-size` | `2` | Number of GPUs to shard the model across. |
| `--gpu-memory-utilization` | `0.8` | Fraction of GPU memory vLLM may use for weights and KV cache. |
| `--tool-call-parser` | `glm47` | Server-side parser that emits structured `tool_calls` on the response. |
| `--enable-auto-tool-choice` | (no value) | Let the model choose when to call tools without requiring `tool_choice: "required"`. |
| `--trust-remote-code` | (no value) | Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures). |
| `--max-model-len` | `auto` | Maximum context length (tokens) the server accepts per request. |
## Deploy
Push the config to Baseten:
```sh theme={"system"}
uvx truss push
```
You should see output similar to:
```text theme={"system"}
✨ Model glm-4.7-flash-latency was successfully pushed ✨
🪵 View logs for your deployment at https://app.baseten.co/models/abcd1234/logs/wxyz5678
```
Your **model ID** is the string after `/models/` in the logs URL (`abcd1234` in the example). Use it wherever you see `{model_id}` in the next section.
## Call the model
Your deployment serves an OpenAI-compatible API. Replace `{model_id}` with your model ID and make sure `BASETEN_API_KEY` is set.
Now call your deployment to run inference:
```python main.py theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["BASETEN_API_KEY"],
base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)
response = client.chat.completions.create(
model="zai-org/GLM-4.7-Flash",
messages=[
{"role": "user", "content": "What is machine learning?"}
],
)
print(response.choices[0].message.content)
```
```sh theme={"system"}
curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $BASETEN_API_KEY" \
-d '{
"model": "zai-org/GLM-4.7-Flash",
"messages": [
{"role": "user", "content": "What is machine learning?"}
]
}'
```
To let the model call tools, pass a `tools` array. The server returns structured `tool_calls` on the response:
```python theme={"system"}
tools = [{
"type": "function",
"function": {
"name": "get_weather",
"parameters": {
"type": "object",
"properties": {"location": {"type": "string"}},
"required": ["location"],
},
},
}]
response = client.chat.completions.create(
model="zai-org/GLM-4.7-Flash",
messages=[
{"role": "user", "content": "What's the weather in Paris?"}
],
tools=tools,
)
print(response.choices[0].message.tool_calls)
```