Skip to main content
Lookahead decoding is a speculative decoding technique that provides 2x-4x faster inference for suitable workloads by predicting future tokens using n-gram patterns. It’s particularly effective for coding agents and content with predictable patterns.
Lookahead decoding is the Engine-Builder-LLM (v1) speculative path. For MoE and large models on BIS-LLM (v2), use BIS-LLM speculative decoding with Eagle, MTP, or NGram instead.

​
Overview

Lookahead decoding identifies n-gram patterns in the input context and past tokens, speculates on future tokens by generating candidate sequences, verifies those predictions against the model’s actual output, and accepts the verified tokens in a single step. The model still produces every token: it accepts the longest run of guessed tokens that matches its own output, and at the first mismatch it keeps that prefix and falls back to its own next token. The output is identical to decoding token by token: the accepted tokens are exactly what the model would have produced on its own, so speculative decoding changes only how many tokens clear per pass, not the result. The drafted run length depends on lookahead_ngram_size, lookahead_windows_size, and lookahead_verification_set_size, documented under Configuration parameters. The technique works with any model compatible with Engine-Builder-LLM. Baseten’s B10 Lookahead implementation searches up to 10M past tokens for n-gram matches across language patterns.

​
When to use lookahead decoding

Lookahead decoding excels at code generation where programming language syntax creates predictable patterns, and function signatures, variable names, and common idioms all benefit. It also accelerates prompt lookup scenarios where you provide example completions in the prompt, and general low-latency use cases where you can trade slightly decreased throughput for faster individual responses.

​
Limitations

  • Lookahead decoding runs on any GPU that supports TensorRT-LLM (T4 and V100 are not supported). There is no lookahead-specific GPU restriction beyond that.
  • During speculative decoding, sampling is disabled and temperature is set to 0.0.
  • Speculative decoding does not affect output quality. The output depends only on model weights and prompt.
  • Speculative decoding generates multiple tokens at a time. Structured output (xgrammar, outlines) with state-machine guarantees (enforced json through response_format) isn’t possible when lookahead decoding is enabled. Structured outputs are supported in standard Engine-Builder-LLM deployments without speculative decoding.
  • Chunked prefill isn’t supported with lookahead decoding. Baseten disables it automatically when lookahead is enabled.

​
Configuration

​
Basic configuration

Add a speculator section to your build configuration:

​
Configuration parameters

​
speculative_decoding_mode
string
required
Set to LOOKAHEAD_DECODING to enable Baseten’s lookahead decoding algorithm.
​
speculative_decoding_mode: DRAFT_TOKENS_EXTERNAL
string
deprecated
External draft speculation, discontinued in favor of LOOKAHEAD_DECODING. Setting it fails the build with an error directing you to switch. For model-based speculation (Eagle, MTP), use BIS-LLM speculative decoding instead; these methods aren’t available on Engine-Builder-LLM.
​
lookahead_ngram_size
integer
required
Size of n-gram patterns for speculation. Minimum: 1, with no fixed maximum, though the draft-token total computed from the three lookahead sizes must stay at or below 2048 or the build fails. Use 4 for simple patterns, 8 for general use, or 16-32 for complex, highly predictable patterns.
​
lookahead_verification_set_size
integer
required
Size of the verification buffer for speculation. Minimum: 1. Use 1 for high-confidence patterns, 3 for general use, or 5 for complex patterns requiring more verification.
​
lookahead_windows_size
integer
required
Size of the speculation window. Minimum: 1. Pair it with lookahead_verification_set_size for your workload, as in the table below.
​
enable_b10_lookahead
boolean
default:"false"
Enable Baseten’s optimized lookahead algorithm. Set it to true to use Baseten’s B10 lookahead, recommended for the configurations on this page.
Start from the row that matches your workload, then adjust based on the monitoring metrics. Copy the block for your workload into the speculator section of your build configuration:
For dynamic-length speculation, set enable_b10_lookahead: true with both lookahead_windows_size and lookahead_verification_set_size at 1. The speculated length then tracks the quality of the n-gram match: an n-gram of k tokens for a k-token suffix match. The three sizes together determine how many draft tokens the engine generates per step. Keep that total at or below 512 to avoid a performance warning. The engine’s recommended balanced config is:

​
Performance

Lookahead decoding performs best at small batch sizes, so keep concurrency low (1-8 requests) for the largest benefit and set max_batch_size to 32 or 64. Reduce the batch size if the speedup diminishes under load. Lookahead adds no additional GPU memory.

​
Monitoring and troubleshooting

​
Monitoring

Track tokens per second with and without lookahead to measure the speedup, and verification accuracy to see how often speculations succeed. If the speedup diminishes, reduce the batch size; adjust window size by content predictability and n-gram size by verification accuracy.

​
Troubleshooting

Low speed improvement:
  • Check that the content is suitable for lookahead decoding.
  • Reduce the batch size.
  • Adjust the window and n-gram sizes.
Blackwell support: Lookahead isn’t fully supported on Engine-Builder-LLM. See the BIS-LLM overview for Blackwell support.