> ## Documentation Index > Fetch the complete documentation index at: https://docs.baseten.co/llms.txt > Use this file to discover all available pages before exploring further. # Ling 3.0 Flash > InclusionAI Ling 3.0 Flash is a hybrid-linear sparse mixture-of-experts reasoning model with 124B total and 5.1B active parameters.
Reasoning Tool calling Long context
InclusionAI Ling 3.0 Flash is a hybrid-linear sparse mixture-of-experts reasoning model with 124B total and 5.1B active parameters. This preset serves the groupwise INT4 checkpoint on two H100 GPUs through vLLM with a 256K context window, native reasoning output, and automatic tool calling. ## Setup Sign in to Baseten with Truss, then install the OpenAI SDK. **Sign in to Baseten** ```sh theme={"system"} uvx truss login --browser ``` **Install the OpenAI SDK** ```sh theme={"system"} uv pip install openai ``` This preset serves Ling 3.0 Flash INT4 on H100:2 with groupwise INT4 weights and tensor parallel size 2, optimized for low-latency interactive reasoning and tool-calling workloads at the model's native 256K context. H100 × 2 vLLM (8bd082c274fae025... build) 256K 32 ## Write the config Create and move into the project directory: ```sh theme={"system"} mkdir ling-3.0-flash-int4-latency && cd ling-3.0-flash-int4-latency ``` Then create a file named `config.yaml` and paste the following: ```yaml config.yaml theme={"system"} model_name: "model:ling-3.0-flash-int4 preset:latency" model_metadata: description: >- InclusionAI Ling 3.0 Flash INT4 is a 124B-parameter, 5.1B-active hybrid-linear MoE reasoning model served with vLLM on two H100 GPUs. It exposes an OpenAI-compatible chat endpoint with a 256K context window, native reasoning output, and automatic tool calling. repo_id: inclusionAI/Ling-3.0-flash-int4 tags: - openai-compatible example_model_input: model: inclusionAI/Ling-3.0-flash-int4 messages: - role: user content: "Explain why the sky appears blue in three concise sentences." stream: true max_tokens: 512 temperature: 0.6 top_p: 0.95 top_k: 20 chat_template_kwargs: enable_thinking: true base_image: # Ling 3 support (model, tool parser, and reasoning parser) landed after # vLLM v0.27.1. Pin the first available immutable nightly commit that includes it. image: "vllm/vllm-openai:nightly-5a4c8d99242e9e069b604d0e9b969e77f7dd501d@sha256:8bd082c274fae025b7079498fe1da65182ba1d4c2188c0f5a68c1042c38c3695" weights: - source: "hf://inclusionAI/Ling-3.0-flash-int4@959d3a48cf05d2daf5fe7bdbbd3a6bb119e359f2" mount_location: "/app/checkpoint/model" auth_secret_name: "hf_access_token" secrets: hf_access_token: null environment_variables: VLLM_LOGGING_LEVEL: WARNING VLLM_ENGINE_READY_TIMEOUT_S: "3600" docker_server: start_command: >- vllm serve /app/checkpoint/model --host 0.0.0.0 --port 8000 --served-model-name inclusionAI/Ling-3.0-flash-int4 --max-model-len 262144 --trust-remote-code --dtype bfloat16 --gpu-memory-utilization 0.9 --enable-chunked-prefill --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' --enable-prefix-caching --tensor-parallel-size 2 --enable-auto-tool-choice --tool-call-parser ling3 --reasoning-parser ling3 readiness_endpoint: /health liveness_endpoint: /health predict_endpoint: /v1/chat/completions server_port: 8000 resources: accelerator: H100:2 use_gpu: true runtime: predict_concurrency: 32 health_checks: restart_check_delay_seconds: 1800 restart_threshold_seconds: 1200 stop_traffic_threshold_seconds: 120 ``` This preset deploys the groupwise INT4 checkpoint of Ling 3.0 Flash through vLLM on two H100 GPUs with tensor parallelism. It enables prefix caching, chunked prefill, and full plus piecewise CUDA graphs. The deployment exposes an OpenAI-compatible chat completions endpoint with reasoning and tool calling at the model's 256K context window. ## Flags The `start_command` passes these flags to the engine. Each one controls a runtime or serving behavior: | Flag | Value | What it does | | --------------------------- | ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | `--max-model-len` | `262144` | Maximum context length (tokens) the server accepts per request. | | `--trust-remote-code` | (no value) | Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures). | | `--dtype` | `bfloat16` | Weight precision loaded at runtime. **bfloat16:** BF16 weights, no quantization. | | `--gpu-memory-utilization` | `0.9` | Fraction of GPU memory vLLM may use for weights and KV cache. | | `--enable-chunked-prefill` | (no value) | Process long prompts in chunks so decode requests keep running. | | `--compilation-config` | `{"cudagraph_mode":"FULL_AND_PIECEWISE"}` | vLLM compilation passes (op fusion, dead-code elimination). | | `--enable-prefix-caching` | (no value) | Reuse KV cache across requests that share a prefix. | | `--tensor-parallel-size` | `2` | Number of GPUs to shard the model across. | | `--enable-auto-tool-choice` | (no value) | Let the model choose when to call tools without requiring `tool_choice: "required"`. | | `--tool-call-parser` | `ling3` | Server-side parser that emits structured `tool_calls` on the response. **ling3:** Ling 3.0 Flash tool format. | | `--reasoning-parser` | `ling3` | Server-side parser that separates reasoning output into `reasoning_content`. **ling3:** Ling 3.0 Flash thinking format. | ## Deploy Push the config to Baseten: ```sh theme={"system"} uvx truss push ``` You should see output similar to: ```output theme={"system"} ✨ Model ling-3.0-flash-int4-latency was successfully pushed ✨ Model ID: abc1d2ef Deployment ID: xyz123 Endpoint: model-abc1d2ef.api.baseten.co Logs: https://app.baseten.co/models/abc1d2ef/logs/xyz123 ``` `truss push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable. ## Call the model Your deployment serves an OpenAI-compatible API. Now call your deployment to run inference: ```python main.py theme={"system"} import os from openai import OpenAI client = OpenAI( api_key=os.environ["BASETEN_API_KEY"], base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1", ) response = client.chat.completions.create( model="inclusionAI/Ling-3.0-flash-int4", messages=[ {"role": "user", "content": "What is machine learning?"} ], ) print(response.choices[0].message.content) ``` ```sh theme={"system"} curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{ "model": "inclusionAI/Ling-3.0-flash-int4", "messages": [ {"role": "user", "content": "What is machine learning?"} ] }' ``` The server parses the model's chain of thought into a separate `reasoning_content` field on the response. Read it alongside the final answer: ```python theme={"system"} response = client.chat.completions.create( model="inclusionAI/Ling-3.0-flash-int4", messages=[ {"role": "user", "content": "How many r's in strawberry?"} ], ) print(response.choices[0].message.reasoning_content) # chain of thought print(response.choices[0].message.content) # final answer ``` To let the model call tools, pass a `tools` array. The server returns structured `tool_calls` on the response: ```python theme={"system"} tools = [{ "type": "function", "function": { "name": "get_weather", "parameters": { "type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"], }, }, }] response = client.chat.completions.create( model="inclusionAI/Ling-3.0-flash-int4", messages=[ {"role": "user", "content": "What's the weather in Paris?"} ], tools=tools, ) print(response.choices[0].message.tool_calls) ``` ## Next steps Endpoint anatomy, authentication, and sync versus async inference Scale replicas with traffic, including scale to zero