Deploy a model with Eagle speculative decoding

Eagle speculative decoding pairs a model with a lightweight draft head that proposes several tokens at once. The main model verifies the whole run in a single forward pass and accepts the tokens that match, so generation clears more tokens per step without changing the output. On BIS-LLM (v2), you enable it by mounting an Eagle checkpoint alongside the base model and setting a speculative_config block. This tutorial deploys Qwen3-4B-Instruct-2507 with the Qwen3-4B Eagle3 draft head, from a config file to a production API. You’ll set up Truss, write the config, deploy to Baseten, and call the model’s OpenAI-compatible endpoint. For how the speculation loop works and the full parameter reference, see Speculative decoding for BIS-LLM. Before you begin, sign up or sign in to Baseten, then install uv, a fast Python package manager.

Install Truss

uv tool install truss

Sign in

truss login --browser

Prefer not to install? Use uvx truss … for the commands in this tutorial without a permanent install.

Write the config

BIS-LLM deployments need only a config.yaml. Create one that mounts the base model and the Eagle draft head, then turns on Eagle speculation in the speculative_config block:

Your Baseten representative provides the current bis_llm.version and gpuTRTImage values for your workspace. The placeholders below stand in for the values they give you.

config.yaml

model_name: qwen3-4b-eagle
resources:
  accelerator: H100:1
  use_gpu: true
weights:
  - source: hf://Qwen/Qwen3-4B-Instruct-2507
    mount_location: /models/base
  - source: hf://andyjjrt/Qwen3-4B-Instruct-2507-Eagle3
    mount_location: /models/eagle
bis_llm:
  version: "<version from your Baseten representative>"
  config:
    gpuTRTImage: "<image from your Baseten representative>"
    model_name: Qwen/Qwen3-4B-Instruct-2507
    checkpoint_name: Qwen/Qwen3-4B-Instruct-2507
    tensor_parallel_size: 1
    max_num_tokens: 2048
    max_batch_size: 8
    max_seq_len: 4096
    trust_remote_code: true
    backend: pytorch
    enable_chunked_prefill: true
    speculative_config:
      decoding_type: Eagle
      speculative_model_dir: /models/eagle
      max_draft_len: 4
    kv_cache_config:
      free_gpu_memory_fraction: 0.95
      enable_block_reuse: true
  additional_autoscaling_config:
    metrics:
      - name: in_flight_tokens
        target: 6000

weights mounts the base model to /models/base and the Eagle draft head to /models/eagle. Baseten mirrors both to the Baseten Delivery Network for fast cold starts. Qwen3-4B and this Eagle head are ungated, so no access token is needed.
bis_llm selects the BIS-LLM (v2) inference stack. The config block holds the engine and runtime settings.
speculative_config enables Eagle. speculative_model_dir points at the mounted draft head, and max_draft_len sets how many tokens it proposes per step. Raise it for more aggressive speculation, lower it if acceptance drops. See the parameter reference.
additional_autoscaling_config scales on in_flight_tokens rather than request count, which suits mixed-length prompts. See Autoscaling BIS-LLM.

Deploy

Push the model to Baseten:

truss push

You should see:

✨ Model qwen3-4b-eagle was successfully pushed ✨
🪵  View logs for your deployment at https://app.baseten.co/models/abc1d2ef/logs/xyz123

The logs URL contains your model ID, the string after /models/ (for example, abc1d2ef). You’ll need it to call the model.

BIS-LLM deployments take roughly 15-20 minutes to reach Active. The engine loads the base model and Eagle head, then runs CUDA-graph warmup before the deployment passes its readiness check. Requests sent before then return Model is not ready.

Call the model

Once the deployment shows Active in the dashboard, call its OpenAI-compatible endpoint. Speculative decoding runs greedily, so set temperature to 0. Replace {model_id} with your model ID.

Python
cURL

Install the OpenAI SDK if you don’t have it:

uv pip install openai

Send a request:

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="Qwen/Qwen3-4B-Instruct-2507",
    messages=[
        {"role": "system", "content": "You are a Python programming assistant. Write clean, efficient code."},
        {"role": "user", "content": "Write FizzBuzz in Python."},
    ],
    temperature=0,
)

print(response.choices[0].message.content)

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "Qwen/Qwen3-4B-Instruct-2507",
    "messages": [
      {"role": "system", "content": "You are a Python programming assistant. Write clean, efficient code."},
      {"role": "user", "content": "Write FizzBuzz in Python."}
    ],
    "temperature": 0
  }'

The response is identical to what the model would produce without speculation. To see how often the draft head’s tokens are accepted, watch the speculation_rate metric on the BIS-LLM dashboard.

Next steps

Speculative decoding for BIS-LLM

Eagle, MTP, and N-gram parameters, and the acceptance-rate metric to watch.

BIS-LLM configuration

Full config.yaml reference for the v2 inference stack.

Lookahead decoding (v1)

N-gram speculation on Engine-Builder-LLM, the self-serve v1 path.

Autoscaling BIS-LLM

Token-based autoscaling for prefill, decode, and aggregated replicas.

Examples

Models

Engines

Custom Docker servers

Custom Python models

Chains

Deploy a model with Eagle speculative decoding

Write the config

Deploy

Call the model

Next steps

Speculative decoding for BIS-LLM

BIS-LLM configuration

Lookahead decoding (v1)

Autoscaling BIS-LLM

​Install and sign in

​Write the config

​Deploy

​Call the model

​Next steps

Speculative decoding for BIS-LLM

BIS-LLM configuration

Lookahead decoding (v1)

Autoscaling BIS-LLM

Install and sign in

Write the config

Deploy

Call the model

Next steps