> ## Documentation Index
> Fetch the complete documentation index at: https://docs.baseten.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Deploy a model with Eagle speculative decoding

> Deploy Qwen3-4B on BIS-LLM (v2) with an Eagle draft head for faster generation, then call it from an OpenAI-compatible endpoint.

export const BisLlmEnterpriseGate = ({feature, plural = false}) => {
  if (!feature) {
    return <p>
        Some BIS-LLM features are available on Enterprise plans only.{" "}
        <a href="mailto:support@baseten.co">Contact us</a>{" "}
        to enable BIS-LLM or its Enterprise features.
      </p>;
  }
  return <p>
      {feature} {plural ? "are Enterprise features on BIS-LLM" : "is an Enterprise feature on BIS-LLM"}.{" "}
      <a href="mailto:support@baseten.co">Contact us</a>{" "}
      to enable {plural ? "them" : "it"} for your deployment.
    </p>;
};

Eagle speculative decoding pairs a model with a lightweight draft head that proposes several tokens at once. The main model verifies the whole run in a single forward pass and accepts the tokens that match, so generation clears more tokens per step without changing the output. On [BIS-LLM (v2)](/engines/bis-llm/overview), you enable it by mounting an Eagle checkpoint alongside the base model and setting a `speculative_config` block.

This tutorial deploys [Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507) with the [Qwen3-4B Eagle3](https://huggingface.co/andyjjrt/Qwen3-4B-Instruct-2507-Eagle3) draft head, from a config file to a production API. You'll set up the Baseten CLI, write the config, deploy to Baseten, and call the model's OpenAI-compatible endpoint. For how the speculation loop works and the full parameter reference, see [Speculative decoding for BIS-LLM](/engines/bis-llm/advanced-features#speculative-decoding).

<BisLlmEnterpriseGate feature="BIS-LLM and Eagle speculative decoding" plural />

## Install and sign in

Before you begin, [sign up](https://app.baseten.co/signup) or [sign in](https://app.baseten.co/login) to Baseten, then install the Baseten CLI. Browser login opens a tab to approve this device, so there's no API key to copy or paste.

<Columns cols={2}>
  <Column>
    **Install the Baseten CLI**

    <Tabs>
      <Tab title="macOS or Linux">
        ```bash Terminal theme={"system"}
        brew tap basetenlabs/baseten
        brew install baseten
        ```
      </Tab>

      <Tab title="Windows">
        Download and extract the binary, then move `baseten.exe` to a directory on your `PATH`:

        ```powershell Terminal theme={"system"}
        Invoke-WebRequest `
          https://github.com/basetenlabs/baseten-cli/releases/download/v0.4.0/baseten_0.4.0_windows_amd64.zip `
          -OutFile baseten.zip; Expand-Archive -Force baseten.zip .
        ```
      </Tab>
    </Tabs>

    For other platforms or a specific version, see the [Baseten CLI install reference](/reference/cli/baseten/overview#install).
  </Column>

  <Column>
    **Sign in**

    ```sh theme={"system"}
    baseten auth login
    ```
  </Column>
</Columns>

<Tip>
  Prefer the Truss CLI? Use `uvx truss login --browser` and `uvx truss …` for the commands in this tutorial without a permanent install.
</Tip>

## Write the config

BIS-LLM deployments need only a `config.yaml`. Create one that mounts the base model and the Eagle draft head, then turns on Eagle speculation in the `speculative_config` block:

<Note>
  Your Baseten representative provides the current `bis_llm.version` and `gpuTRTImage` values for your workspace. The placeholders below stand in for the values they give you.
</Note>

```yaml config.yaml theme={"system"}
model_name: qwen3-4b-eagle
resources:
  accelerator: H100:1
  use_gpu: true
weights:
  - source: hf://Qwen/Qwen3-4B-Instruct-2507
    mount_location: /models/base
  - source: hf://andyjjrt/Qwen3-4B-Instruct-2507-Eagle3
    mount_location: /models/eagle
bis_llm:
  version: "<version from your Baseten representative>"
  config:
    gpuTRTImage: "<image from your Baseten representative>"
    model_name: Qwen/Qwen3-4B-Instruct-2507
    checkpoint_name: Qwen/Qwen3-4B-Instruct-2507
    tensor_parallel_size: 1
    engine_config:
      backend: pytorch
      trust_remote_code: true
      enable_chunked_prefill: true
      max_num_tokens: 2048
      max_batch_size: 8
      max_seq_len: 4096
      speculative_config:
        decoding_type: Eagle
        speculative_model_dir: /models/eagle
        max_draft_len: 4
      kv_cache_config:
        free_gpu_memory_fraction: 0.95
        enable_block_reuse: true
  additional_autoscaling_config:
    metrics:
      - name: in_flight_tokens
        target: 6000
```

* `weights` mounts the base model to `/models/base` and the Eagle draft head to `/models/eagle`. Baseten mirrors both to the [Baseten Delivery Network](/development/model/bdn) for fast cold starts. Qwen3-4B and this Eagle head are ungated, so no access token is needed.
* `bis_llm` selects the [BIS-LLM (v2)](/engines/bis-llm/overview) inference stack. The `config` block holds the engine and runtime settings.
* `engine_config.speculative_config` enables Eagle. `speculative_model_dir` points at the mounted draft head, and `max_draft_len` sets how many tokens it proposes per step. Raise it for more aggressive speculation, lower it if acceptance drops. See [the parameter reference](/engines/bis-llm/advanced-features#speculative-decoding).
* `additional_autoscaling_config` scales on `in_flight_tokens` rather than request count, which suits mixed-length prompts. See [Autoscaling BIS-LLM](/engines/performance-concepts/autoscaling-engines#bis-llm).

## Deploy

Push the model to Baseten:

```sh theme={"system"}
baseten model push
```

You should see:

```output theme={"system"}
Pushing model "qwen3-4b-eagle"...
Uploading model...
Uploaded model in 0s
✨ Model qwen3-4b-eagle was successfully pushed ✨

  Model:       qwen3-4b-eagle (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef
```

<Note>
  BIS-LLM deployments take roughly 15-20 minutes to reach **Active**. The engine loads the base model and Eagle head, then runs CUDA-graph warmup before the deployment passes its readiness check. Requests sent before then return `Model is not ready`.
</Note>

## Call the model

Once the deployment shows **Active** in the dashboard, call its OpenAI-compatible endpoint. Speculative decoding runs greedily, so set `temperature` to `0`. Replace `{model_id}` with your model ID.

<Tabs>
  <Tab title="Python">
    Install the OpenAI SDK if you don't have it:

    ```sh theme={"system"}
    uv pip install openai
    ```

    Send a request:

    ```python theme={"system"}
    import os
    from openai import OpenAI

    client = OpenAI(
        api_key=os.environ["BASETEN_API_KEY"],
        base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
    )

    response = client.chat.completions.create(
        model="Qwen/Qwen3-4B-Instruct-2507",
        messages=[
            {"role": "system", "content": "You are a Python programming assistant. Write clean, efficient code."},
            {"role": "user", "content": "Write FizzBuzz in Python."},
        ],
        temperature=0,
    )

    print(response.choices[0].message.content)
    ```
  </Tab>

  <Tab title="cURL">
    ```sh theme={"system"}
    curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $BASETEN_API_KEY" \
      -d '{
        "model": "Qwen/Qwen3-4B-Instruct-2507",
        "messages": [
          {"role": "system", "content": "You are a Python programming assistant. Write clean, efficient code."},
          {"role": "user", "content": "Write FizzBuzz in Python."}
        ],
        "temperature": 0
      }'
    ```
  </Tab>
</Tabs>

The response is identical to what the model would produce without speculation. To see how often the draft head's tokens are accepted, watch the `speculation_rate` metric on the BIS-LLM dashboard.

## Next steps

<CardGroup cols={2}>
  <Card title="Speculative decoding for BIS-LLM" icon="bolt" href="/engines/bis-llm/advanced-features#speculative-decoding">
    Eagle, MTP, and N-gram parameters, and the acceptance-rate metric to watch.
  </Card>

  <Card title="BIS-LLM configuration" icon="gear" href="/engines/bis-llm/bis-llm-config">
    Full `config.yaml` reference for the v2 inference stack.
  </Card>

  <Card title="Lookahead decoding (v1)" icon="gauge-high" href="/engines/engine-builder-llm/lookahead-decoding">
    N-gram speculation on Engine-Builder-LLM, the self-serve v1 path.
  </Card>

  <Card title="Autoscaling BIS-LLM" icon="chart-line" href="/engines/performance-concepts/autoscaling-engines#bis-llm">
    Token-based autoscaling for prefill, decode, and aggregated replicas.
  </Card>
</CardGroup>