Skip to main content
BIS-LLM (Baseten Inference Stack v2) is the engine for Mixture of Experts (MoE) models and large dense LLMs. It targets MoE families (DeepSeek V3.x, Qwen3MoE, Kimi-K2, Llama 4, GLM-4.7, GPT-OSS 120B) and the largest dense models, where the standard request-based autoscaler and a single-server inference engine both leave performance on the table. The v2 stack adds token-based autoscaling, KV-aware routing, speculative decoding, expert parallel load balancing, and DP attention. Deployments mirror build artifacts to the Baseten Delivery Network so cold starts stay fast.

BIS-LLM features

BIS-LLM adds token-based autoscaling, KV-aware routing, and speculative decoding to the standard inference path. For more information, see Autoscaling engines and Advanced features for BIS-LLM.

Token-based autoscaling

Scales replicas on target_in_flight_tokens rather than request concurrency, so mixed-length prompt workloads scale on real compute load.

KV-aware routing

Routes requests to the worker most likely to serve them from KV cache. Lower time-to-first-token on prefix-overlapping traffic.

Speculative decoding

Eagle, MTP, and N-gram speculation. Multiple tokens per forward pass on supported architectures.
Disaggregated serving is a limited-availability option for selected workloads, not the default BIS-LLM topology. For more information, see Disaggregated serving.

A canonical configuration

BIS-LLM is configured in the bis_llm block of config.yaml. A prequantized DeepSeek V3.2 deployment on H200 looks like:
config.yaml
After baseten model push, the BDN mirrors weights to GPU-local storage and the deployment exposes OpenAI-compatible /v1/chat/completions and /v1/responses, plus Anthropic-compatible /v1/messages. For more information, see BIS-LLM configuration. For tuning advice on a specific or fine-tuned model, contact us.

OpenAI-compatible inference

BIS-LLM deployments expose OpenAI-compatible /v1/chat/completions, /v1/completions, /v1/responses, and /v1/embeddings (where applicable), plus Anthropic-compatible /v1/messages. OpenAI and Anthropic client SDKs can call the matching routes without modification. Set model to the model_name from your bis_llm config. The deployment returns 404 for a model name it doesn’t serve:

Responses API

Call the same deployment with the OpenAI Responses API, for example from the OpenAI Agents SDK.
Requests are stateless: send the full conversation as input on every call. For streaming, function tools, structured output, and unsupported fields, see Responses. Structured outputs and function calling are supported through the standard OpenAI parameters and have their own reference pages.

Observability

BIS-LLM emits metrics from three components. Each has its own dashboard section: Engine-level metrics, available on every BIS-LLM deployment: For more information about how tps_per_request, decode_tokens_per_second, and total output throughput differ, see LLM token speed. If you run Enterprise features, add kv_cache_hit_rate (KV-aware routing, in the router domain) or speculation_rate (Eagle/MTP) next. For more information about the configuration that produces speculation_rate, see Speculative decoding.

Migrate from Engine-Builder-LLM

The Engine-Builder trt_llm build schema has a v2 (inference_stack: v2) that moves runtime fields out of build:, renames tensor_parallel_count to tensor_parallel_size, and drops fields v2 handles automatically (plugin_configuration, base_model). For more information, see Migrate from Engine-Builder-LLM.