> ## Documentation Index > Fetch the complete documentation index at: https://docs.baseten.co/llms.txt > Use this file to discover all available pages before exploring further. # GLM-4.7 > GLM-4.7 recipes: 2 variants (Standard, Flash), MoE architecture.
Reasoning Tool calling Long context
## Setup Install the Baseten CLI and sign in, then install the OpenAI SDK. **Install and sign in to Baseten** ```bash Terminal theme={"system"} brew tap basetenlabs/baseten brew trust --formula basetenlabs/baseten/baseten brew install baseten ``` Download and extract the Windows archive for the latest Baseten CLI release: ```powershell Terminal theme={"system"} $release = Invoke-RestMethod https://api.github.com/repos/basetenlabs/baseten-cli/releases/latest $asset = $release.assets | Where-Object { $_.name -like "*_windows_amd64.zip" } Invoke-WebRequest $asset.browser_download_url -OutFile baseten.zip Expand-Archive -Force baseten.zip . ``` Move `baseten.exe` to a directory on your `PATH`. For other platforms or a specific version, see the [Baseten CLI install reference](/reference/cli/baseten/overview#install). ```sh theme={"system"} baseten auth login ``` **Install the OpenAI SDK** ```sh theme={"system"} uv pip install openai ``` To install without Homebrew, see the [Baseten CLI install reference](/reference/cli/baseten/overview#install). Pick the model you want to deploy. Each tab is a self-contained recipe. [zai-org/GLM-4.7](https://huggingface.co/zai-org/GLM-4.7) is a MoE model with up to 198K context. This preset serves GLM-4.7 from an FP4 checkpoint on B200:4, delivering frontier-class reasoning at single-node cost. B200 × 4 TRT-LLM v2 198K 64 ## Write the config Create and move into the project directory: ```sh theme={"system"} mkdir glm-4.7-latency && cd glm-4.7-latency ``` Then create a file named `config.yaml` and paste the following: ```yaml config.yaml theme={"system"} model_name: "model:glm-4.7 preset:latency" resources: accelerator: B200:4 cpu: "1" memory: 10Gi use_gpu: true model_metadata: tags: - openai-compatible example_model_input: model: glm47 messages: - role: user content: "Given an array of integers nums and an integer target, return indices of the two numbers such that they add up to target. You may assume that each input would have exactly one solution, and you may not use the same element twice. You can return the answer in any order. class Solution: def twoSum(self, nums: List[int], target: int) -> List[int]:" stream: true max_tokens: 2048 temperature: 0.5 trt_llm: build: checkpoint_repository: repo: baseten-admin/glm-4.7-fp4 revision: main source: HF # runtime_secret_name: "hf_access_token_custom" inference_stack: v2 runtime: enable_chunked_prefill: true max_batch_size: 64 max_num_tokens: 8192 max_seq_len: 202752 tensor_parallel_size: 4 served_model_name: glm47 patch_kwargs: disable_overlap_scheduler: True moe_expert_parallel_size: 4 moe_config: use_low_precision_moe_combine: true backend: TRTLLM kv_cache_config: free_gpu_memory_fraction: 0.8 enable_block_reuse: true cuda_graph_config: enable_padding: true max_batch_size: 64 speculative_config: decoding_type: MTP num_nextn_predict_layers: 3 autotuner_enabled: false reasoning_parser: glm47 tool_call_parser: glm47 ``` ## Key parameters [Baseten Inference Stack](/engines/bis-llm/overview) (BIS) reads these fields from the `trt_llm` block. Each one shapes how the engine is built and served: | Parameter | Value | | - | - | | Tensor parallel size | `4` | | Max sequence length | `202752` | | Max batch size | `64` | | Max batched tokens | `8192` | | Chunked prefill | `enabled` | | Inference stack | `v2` | | Served model name | `glm47` | ## Deploy Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it: ```sh Baseten CLI theme={"system"} baseten model push ``` ```sh Truss CLI theme={"system"} uvx truss push ``` You should see output similar to: ```output theme={"system"} Pushing model "glm-4.7-latency"... Uploading model... Uploaded model in 0s ✨ Model glm-4.7-latency was successfully pushed ✨ Model: glm-4.7-latency (abc1d2ef) Deployment: xyz123 Environment: production 🪵 View logs: deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123 environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed) app: https://app.baseten.co/models/abc1d2ef/logs/xyz123 🚀 Invoke your model: URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict CLI: baseten model predict --model-id abc1d2ef ``` `baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable. ## Call the model Your deployment serves an OpenAI-compatible API. Now call your deployment to run inference: ```python main.py theme={"system"} import os from openai import OpenAI client = OpenAI( api_key=os.environ["BASETEN_API_KEY"], base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1", ) response = client.chat.completions.create( model="glm47", messages=[ {"role": "user", "content": "What is machine learning?"} ], ) print(response.choices[0].message.content) ``` ```sh theme={"system"} curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{ "model": "glm47", "messages": [ {"role": "user", "content": "What is machine learning?"} ] }' ``` [zai-org/GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash) is a MoE model with up to 128K context. This preset serves GLM-4.7 Flash on H100:2 with the glm47 tool-call parser enabled, tuned for latency-sensitive agent workflows. H100 × 2 vLLM (0.22.0-cu129 build) 128K 32 ## Write the config Create and move into the project directory: ```sh theme={"system"} mkdir glm-4.7-flash-latency && cd glm-4.7-flash-latency ``` Then create a file named `config.yaml` and paste the following: ```yaml config.yaml theme={"system"} model_name: "model:glm-4.7-flash preset:latency" model_metadata: description: >- Zhipu GLM-4.7 Flash using vLLM (H100 × 2 TP), fast GLM tool calling and auto tool choice from BDN-mounted weights. repo_id: zai-org/GLM-4.7-Flash example_model_input: model: zai-org/GLM-4.7-Flash messages: - role: system content: "You are a helpful assistant." - role: user content: "What is the meaning of life?" stream: true max_tokens: 32768 temperature: 0.7 tags: - openai-compatible base_image: image: vllm/vllm-openai:v0.22.0-cu129 weights: - source: "hf://zai-org/GLM-4.7-Flash@7dd20894a642a0aa287e9827cb1a1f7f91386b67" mount_location: "/app/checkpoint/model" environment_variables: VLLM_LOGGING_LEVEL: WARNING VLLM_ENGINE_READY_TIMEOUT_S: "3600" docker_server: start_command: >- sh -c 'GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) && vllm serve /app/checkpoint/model --tensor-parallel-size $GPU_COUNT --gpu-memory-utilization 0.8 --tool-call-parser glm47 --enable-auto-tool-choice --served-model-name zai-org/GLM-4.7-Flash --host 0.0.0.0 --port 8000 --trust-remote-code --max-model-len auto --enable-prefix-caching --load-format runai_streamer' readiness_endpoint: /health liveness_endpoint: /health predict_endpoint: /v1/chat/completions server_port: 8000 resources: accelerator: H100:2 cpu: "1" memory: 2Gi use_gpu: true runtime: is_websocket_endpoint: false predict_concurrency: 32 transport: kind: http health_checks: startup_threshold_seconds: 1800 restart_threshold_seconds: 1200 stop_traffic_threshold_seconds: 120 ``` ## Flags The `start_command` passes these flags to the engine. Each one controls a runtime or serving behavior: | Flag | Value | What it does | | - | - | - | | `--tensor-parallel-size` | `$GPU_COUNT` | Number of GPUs to shard the model across. | | `--gpu-memory-utilization` | `0.8` | Fraction of GPU memory vLLM may use for weights and KV cache. | | `--tool-call-parser` | `glm47` | Server-side parser that emits structured `tool_calls` on the response. **glm47:** GLM-4.7-style tool format, also used by GLM-5 and GLM-5.3. | | `--enable-auto-tool-choice` | (no value) | Let the model choose when to call tools without requiring `tool_choice: "required"`. | | `--trust-remote-code` | (no value) | Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures). | | `--max-model-len` | `auto` | Maximum context length (tokens) the server accepts per request. | | `--enable-prefix-caching` | (no value) | Reuse KV cache across requests that share a prefix. | | `--load-format` | `runai_streamer` | Weight loading backend. **runai\_streamer:** Stream weights from object storage without materializing to disk. | ## Deploy Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it: ```sh Baseten CLI theme={"system"} baseten model push ``` ```sh Truss CLI theme={"system"} uvx truss push ``` You should see output similar to: ```output theme={"system"} Pushing model "glm-4.7-flash-latency"... Uploading model... Uploaded model in 0s ✨ Model glm-4.7-flash-latency was successfully pushed ✨ Model: glm-4.7-flash-latency (abc1d2ef) Deployment: xyz123 Environment: production 🪵 View logs: deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123 environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed) app: https://app.baseten.co/models/abc1d2ef/logs/xyz123 🚀 Invoke your model: URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict CLI: baseten model predict --model-id abc1d2ef ``` `baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable. ## Call the model Your deployment serves an OpenAI-compatible API. Now call your deployment to run inference: ```python main.py theme={"system"} import os from openai import OpenAI client = OpenAI( api_key=os.environ["BASETEN_API_KEY"], base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1", ) response = client.chat.completions.create( model="zai-org/GLM-4.7-Flash", messages=[ {"role": "user", "content": "What is machine learning?"} ], ) print(response.choices[0].message.content) ``` ```sh theme={"system"} curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{ "model": "zai-org/GLM-4.7-Flash", "messages": [ {"role": "user", "content": "What is machine learning?"} ] }' ``` To let the model call tools, pass a `tools` array. The server returns structured `tool_calls` on the response: ```python theme={"system"} tools = [{ "type": "function", "function": { "name": "get_weather", "parameters": { "type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"], }, }, }] response = client.chat.completions.create( model="zai-org/GLM-4.7-Flash", messages=[ {"role": "user", "content": "What's the weather in Paris?"} ], tools=tools, ) print(response.choices[0].message.tool_calls) ``` ## Next steps Endpoint anatomy, authentication, and sync versus async inference Scale replicas with traffic, including scale to zero