> ## Documentation Index
> Fetch the complete documentation index at: https://docs.baseten.co/llms.txt
> Use this file to discover all available pages before exploring further.
# Speculative decoding
> Lookahead decoding on Engine-Builder-LLM (v1) for code generation and predictable content
Lookahead decoding is a speculative decoding technique that provides 2x-4x faster inference for suitable workloads by predicting future tokens using n-gram patterns. It's particularly effective for coding agents and content with predictable patterns.
Lookahead decoding is the Engine-Builder-LLM (v1) speculative path. For MoE and large models on BIS-LLM (v2), use [BIS-LLM speculative decoding](/engines/bis-llm/advanced-features#speculative-decoding) with Eagle, MTP, or NGram instead.
## Overview
Lookahead decoding identifies n-gram patterns in the input context and past tokens, speculates on future tokens by generating candidate sequences, verifies those predictions against the model's actual output, and accepts the verified tokens in a single step. The model still produces every token: it accepts the longest run of guessed tokens that matches its own output, and at the first mismatch it keeps that prefix and falls back to its own next token.
The output is identical to decoding token by token: the accepted tokens are exactly what the model would have produced on its own, so speculative decoding changes only how many tokens clear per pass, not the result. The drafted run length depends on `lookahead_ngram_size`, `lookahead_windows_size`, and `lookahead_verification_set_size`, documented under [Configuration parameters](#configuration-parameters).
The technique works with any model compatible with Engine-Builder-LLM. Baseten's B10 Lookahead implementation searches up to 10M past tokens for n-gram matches across language patterns.
## When to use lookahead decoding
Lookahead decoding excels at code generation where programming language syntax creates predictable patterns, and function signatures, variable names, and common idioms all benefit. It also accelerates prompt lookup scenarios where you provide example completions in the prompt, and general low-latency use cases where you can trade slightly decreased throughput for faster individual responses.
### Limitations
* Lookahead decoding runs on any GPU that supports TensorRT-LLM (T4 and V100 are not supported). There is no lookahead-specific GPU restriction beyond that.
* During speculative decoding, sampling is disabled and temperature is set to 0.0.
* Speculative decoding does not affect output quality. The output depends only on model weights and prompt.
* Speculative decoding generates multiple tokens at a time. Structured output (xgrammar, outlines) with state-machine guarantees (enforced json through `response_format`) isn't possible when lookahead decoding is enabled. Structured outputs are supported in standard Engine-Builder-LLM deployments without speculative decoding.
* Chunked prefill isn't supported with lookahead decoding. Baseten disables it automatically when lookahead is enabled.
## Configuration
### Basic configuration
Add a `speculator` section to your build configuration:
```yaml theme={"system"}
trt_llm:
build:
base_model: decoder
checkpoint_repository:
source: HF
repo: "Qwen/Qwen3.5-4B"
speculator:
speculative_decoding_mode: LOOKAHEAD_DECODING
lookahead_windows_size: 3
lookahead_ngram_size: 8
lookahead_verification_set_size: 3
enable_b10_lookahead: true
```
### Configuration parameters
Set to `LOOKAHEAD_DECODING` to enable Baseten's lookahead decoding algorithm.
External draft speculation, discontinued in favor of `LOOKAHEAD_DECODING`. Setting it fails the build with an error directing you to switch. For model-based speculation (Eagle, MTP), use [BIS-LLM speculative decoding](/engines/bis-llm/advanced-features#speculative-decoding) instead; these methods aren't available on Engine-Builder-LLM.
Size of n-gram patterns for speculation. Minimum: 1, with no fixed maximum, though the draft-token total computed from the three lookahead sizes must stay at or below 2048 or the build fails. Use `4` for simple patterns, `8` for general use, or `16-32` for complex, highly predictable patterns.
Size of the verification buffer for speculation. Minimum: 1. Use `1` for high-confidence patterns, `3` for general use, or `5` for complex patterns requiring more verification.
Size of the speculation window. Minimum: 1. Pair it with `lookahead_verification_set_size` for your workload, as in the table below.
Enable Baseten's optimized lookahead algorithm. Set it to `true` to use Baseten's B10 lookahead, recommended for the configurations on this page.
### Recommended configurations
Start from the row that matches your workload, then adjust based on the [monitoring](#monitoring-and-troubleshooting) metrics.
| Workload | `lookahead_windows_size` | `lookahead_ngram_size` | `lookahead_verification_set_size` |
| ----------------------------- | ------------------------ | ---------------------- | --------------------------------- |
| General text generation | 3 | 8 | 3 |
| Coding and predictable syntax | 1 | 8 | 3 |
| Highly repetitive content | 1 | 32 | 1 |
Copy the block for your workload into the `speculator` section of your build configuration:
```yaml General theme={"system"}
speculator:
speculative_decoding_mode: LOOKAHEAD_DECODING
lookahead_windows_size: 3
lookahead_ngram_size: 8
lookahead_verification_set_size: 3
enable_b10_lookahead: true
```
```yaml Coding theme={"system"}
speculator:
speculative_decoding_mode: LOOKAHEAD_DECODING
lookahead_windows_size: 1
lookahead_ngram_size: 8
lookahead_verification_set_size: 3
enable_b10_lookahead: true
```
```yaml Highly repetitive theme={"system"}
speculator:
speculative_decoding_mode: LOOKAHEAD_DECODING
lookahead_windows_size: 1
lookahead_ngram_size: 32
lookahead_verification_set_size: 1
enable_b10_lookahead: true
```
For dynamic-length speculation, set `enable_b10_lookahead: true` with both `lookahead_windows_size` and `lookahead_verification_set_size` at `1`. The speculated length then tracks the quality of the n-gram match: an n-gram of `k` tokens for a `k`-token suffix match.
The three sizes together determine how many draft tokens the engine generates per step. Keep that total at or below 512 to avoid a performance warning. The engine's recommended balanced config is:
```yaml theme={"system"}
speculator:
speculative_decoding_mode: LOOKAHEAD_DECODING
lookahead_windows_size: 7
lookahead_ngram_size: 5
lookahead_verification_set_size: 3
enable_b10_lookahead: true
```
## Performance
Lookahead decoding performs best at small batch sizes, so keep concurrency low (1-8 requests) for the largest benefit and set `max_batch_size` to 32 or 64. Reduce the batch size if the speedup diminishes under load. Lookahead adds no additional GPU memory.
## Monitoring and troubleshooting
### Monitoring
Track tokens per second with and without lookahead to measure the speedup, and verification accuracy to see how often speculations succeed. If the speedup diminishes, reduce the batch size; adjust window size by content predictability and n-gram size by verification accuracy.
### Troubleshooting
**Low speed improvement:**
* Check that the content is suitable for lookahead decoding.
* Reduce the batch size.
* Adjust the window and n-gram sizes.
**Blackwell support:** Lookahead isn't fully supported on Engine-Builder-LLM. See the [BIS-LLM overview](/engines/bis-llm/overview) for Blackwell support.
## Related
* [Engine-Builder-LLM overview](/engines/engine-builder-llm/overview): Main engine documentation.
* [Engine-Builder-LLM configuration](/engines/engine-builder-llm/engine-builder-config): Complete reference config.
* [BIS-LLM speculative decoding](/engines/bis-llm/advanced-features#speculative-decoding): Eagle, MTP, and NGram on v2.
* [Structured outputs documentation](/inference/structured-outputs): JSON schema validation.
* [Deploy a model with Eagle speculative decoding](/examples/speculative-decoding): Step-by-step tutorial for the BIS-LLM (v2) Eagle path.