> ## Documentation Index > Fetch the complete documentation index at: https://docs.baseten.co/llms.txt > Use this file to discover all available pages before exploring further. # GLM-5.3 Flash > Z.ai GLM-5.3 Flash is a multimodal mixture-of-experts model with 321B total and 18B active parameters.
Reasoning Tool calling Multimodal (image) Long context
Z.ai GLM-5.3 Flash is a multimodal mixture-of-experts model with 321B total and 18B active parameters. This preset serves the native FP8 checkpoint on eight H100 GPUs through vLLM with a 1M-token context window, image and video inputs, always-on reasoning, automatic tool calling, and multi-token prediction speculative decoding. ## Setup Sign in to Baseten with Truss, then install the OpenAI SDK. **Sign in to Baseten** ```sh theme={"system"} uvx truss login --browser ``` **Install the OpenAI SDK** ```sh theme={"system"} uv pip install openai ``` This preset serves GLM-5.3 Flash on H100:8 from its native FP8 checkpoint, with a multi-token prediction head proposing five speculative tokens per step to speed up decoding. H100 × 8 vLLM (glm53-flash build) 1M 16 ## Write the config Create and move into the project directory: ```sh theme={"system"} mkdir glm-5.3-flash-latency && cd glm-5.3-flash-latency ``` Then create a file named `config.yaml` and paste the following: ```yaml config.yaml theme={"system"} model_name: "model:glm-5.3-flash preset:latency" model_metadata: description: >- Z.ai GLM-5.3-Flash is a 321B-parameter, 18B-active multimodal MoE served from its native FP8 checkpoint with vLLM on eight H100 GPUs. It provides OpenAI-compatible chat completions, reasoning, tool calling, image and video inputs, MTP speculative decoding, and a 1M-token context window. repo_id: zai-org/GLM-5.3-Flash tags: - openai-compatible - multimodal - moe - reasoning - tool-use - fp8 example_model_input: model: zai-org/GLM-5.3-Flash messages: - role: user content: "Explain sparse attention in three concise sentences." stream: true max_tokens: 512 temperature: 1.0 top_p: 0.95 base_image: # Dedicated image required until GLM-5.3-Flash support reaches a stable vLLM release. image: "vllm/vllm-openai:glm53-flash@sha256:2c6da6c6f16ed15c91e412d896dba13701f25fe1861eaec9ddaa4db34d1d21c4" weights: - source: "hf://zai-org/GLM-5.3-Flash@3f1971b7b5f7a528c9c4ef6212c8785298a8c24a" mount_location: "/app/checkpoint/model" auth_secret_name: "hf_access_token" secrets: hf_access_token: null environment_variables: VLLM_LOGGING_LEVEL: WARNING VLLM_ENGINE_READY_TIMEOUT_S: "3600" docker_server: start_command: >- vllm serve /app/checkpoint/model --host 0.0.0.0 --port 8000 --served-model-name zai-org/GLM-5.3-Flash --tensor-parallel-size 8 --max-model-len 1048576 --gpu-memory-utilization 0.90 --max-num-seqs 16 --enable-prefix-caching --limit-mm-per-prompt.image 1 --limit-mm-per-prompt.video 1 --tool-call-parser glm47 --reasoning-parser glm45 --default-chat-template-kwargs.reasoning_effort high --enable-auto-tool-choice --speculative-config.method mtp --speculative-config.num_speculative_tokens 5 --no-enable-flashinfer-autotune readiness_endpoint: /health liveness_endpoint: /health predict_endpoint: /v1/chat/completions server_port: 8000 resources: instance_type: H100:8 runtime: predict_concurrency: 16 health_checks: restart_check_delay_seconds: 1800 restart_threshold_seconds: 1200 stop_traffic_threshold_seconds: 120 ``` The container loads the FP8 checkpoint to `/app/checkpoint/model` and serves the OpenAI-compatible API on port 8000 with tensor parallel size 8 across the eight H100 GPUs. The engine caps each request at 1,048,576 tokens and runs 16 concurrent sequences, keeping weights in FP8 while the KV cache stays in BF16, because the current implementation does not support an FP8 KV cache on Hopper. Requests that omit `reasoning_effort` inherit the server default of `high`, and each prompt accepts one image and one video alongside its text. ## Flags The `start_command` passes these flags to the engine. Each one controls a runtime or serving behavior: | Flag | Value | What it does | | ------------------------------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | `--tensor-parallel-size` | `8` | Number of GPUs to shard the model across. | | `--max-model-len` | `1048576` | Maximum context length (tokens) the server accepts per request. | | `--gpu-memory-utilization` | `0.90` | Fraction of GPU memory vLLM may use for weights and KV cache. | | `--max-num-seqs` | `16` | Maximum number of concurrent sequences in the batch. | | `--enable-prefix-caching` | (no value) | Reuse KV cache across requests that share a prefix. | | `--limit-mm-per-prompt.image` | `1` | Maximum number of image inputs per prompt. | | `--limit-mm-per-prompt.video` | `1` | Maximum number of video inputs per prompt. | | `--tool-call-parser` | `glm47` | Server-side parser that emits structured `tool_calls` on the response. **glm47:** GLM-4.7-style tool format, also used by GLM-5 and GLM-5.3. | | `--reasoning-parser` | `glm45` | Server-side parser that separates reasoning output into `reasoning_content`. **glm45:** GLM-4.5-style thinking format, also used by GLM-5 and GLM-5.3. | | `--default-chat-template-kwargs.reasoning_effort` | `high` | Reasoning effort the chat template applies when a request omits `reasoning_effort`. Clients override it per request. | | `--enable-auto-tool-choice` | (no value) | Let the model choose when to call tools without requiring `tool_choice: "required"`. | | `--speculative-config.method` | `mtp` | Speculative decoding method. **mtp:** Multi-token prediction head speculation. | | `--speculative-config.num_speculative_tokens` | `5` | Number of tokens the draft speculator proposes per step. | | `--no-enable-flashinfer-autotune` | (no value) | Skip the FlashInfer kernel autotuning pass that normally runs at server startup. | ## Deploy Push the config to Baseten: ```sh theme={"system"} uvx truss push ``` You should see output similar to: ```output theme={"system"} ✨ Model glm-5.3-flash-latency was successfully pushed ✨ Model ID: abc1d2ef Deployment ID: xyz123 Endpoint: model-abc1d2ef.api.baseten.co Logs: https://app.baseten.co/models/abc1d2ef/logs/xyz123 ``` `truss push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable. ## Call the model Your deployment serves an OpenAI-compatible API. Now call your deployment to run inference: ```python main.py theme={"system"} import os from openai import OpenAI client = OpenAI( api_key=os.environ["BASETEN_API_KEY"], base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1", ) response = client.chat.completions.create( model="zai-org/GLM-5.3-Flash", messages=[ {"role": "user", "content": "What is machine learning?"} ], ) print(response.choices[0].message.content) ``` ```sh theme={"system"} curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{ "model": "zai-org/GLM-5.3-Flash", "messages": [ {"role": "user", "content": "What is machine learning?"} ] }' ``` The server parses the model's chain of thought into a separate `reasoning_content` field on the response. Read it alongside the final answer: ```python theme={"system"} response = client.chat.completions.create( model="zai-org/GLM-5.3-Flash", messages=[ {"role": "user", "content": "How many r's in strawberry?"} ], ) print(response.choices[0].message.reasoning_content) # chain of thought print(response.choices[0].message.content) # final answer ``` To let the model call tools, pass a `tools` array. The server returns structured `tool_calls` on the response: ```python theme={"system"} tools = [{ "type": "function", "function": { "name": "get_weather", "parameters": { "type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"], }, }, }] response = client.chat.completions.create( model="zai-org/GLM-5.3-Flash", messages=[ {"role": "user", "content": "What's the weather in Paris?"} ], tools=tools, ) print(response.choices[0].message.tool_calls) ``` ## Next steps Endpoint anatomy, authentication, and sync versus async inference Scale replicas with traffic, including scale to zero