Overview

BIS-LLM (Baseten Inference Stack v2) is the engine for Mixture of Experts (MoE) models and large dense LLMs. It targets MoE families (DeepSeek V3.x, Qwen3MoE, Kimi-K2, Llama 4, GLM-4.7, GPT-OSS 120B) and the largest dense models, where the standard request-based autoscaler and a single-server inference engine both leave performance on the table. The v2 stack adds token-based autoscaling, KV-aware routing, speculative decoding, expert parallel load balancing, and DP attention. Deployments mirror build artifacts to the Baseten Delivery Network so cold starts stay fast.

BIS-LLM features

BIS-LLM adds token-based autoscaling, KV-aware routing, and speculative decoding to the standard inference path. For more information, see Autoscaling engines and Advanced features for BIS-LLM.

Token-based autoscaling

Scales replicas on target_in_flight_tokens rather than request concurrency, so mixed-length prompt workloads scale on real compute load.

KV-aware routing

Routes requests to the worker most likely to serve them from KV cache. Lower time-to-first-token on prefix-overlapping traffic.

Speculative decoding

Eagle, MTP, and N-gram speculation. Multiple tokens per forward pass on supported architectures.

Disaggregated serving is a limited-availability option for selected workloads, not the default BIS-LLM topology. For more information, see Disaggregated serving.

A canonical configuration

BIS-LLM is configured in the bis_llm block of config.yaml. A prequantized DeepSeek V3.2 deployment on H200 looks like:

config.yaml

model_name: deepseek-v3-2
resources:
  accelerator: H200:4
  use_gpu: true
bis_llm:
  config:
    engine_backend: vllm
    checkpoint_name: nvidia/DeepSeek-V3.2-NVFP4
    model_name: deepseek-ai/DeepSeek-V3.2
    tensor_parallel_size: 4
    engine_config:
      max_num_seqs: 64
      max_num_batched_tokens: 16384
      max_model_len: 20480
      gpu_memory_utilization: 0.92
      enable_prefix_caching: true
      enable_chunked_prefill: true
      dtype: auto
      trust_remote_code: true
  additional_autoscaling_config:
    metrics:
      - name: in_flight_tokens
        target: 30000

After truss push, the BDN mirrors weights to GPU-local storage and the deployment exposes OpenAI-compatible /v1/chat/completions. For more information, see BIS-LLM configuration. For tuning advice on a specific or fine-tuned model, contact us.

OpenAI-compatible inference

BIS-LLM deployments expose /v1/chat/completions, /v1/completions, and /v1/embeddings (where applicable). Standard OpenAI client SDKs work without modification:

from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-xxxxxx.api.baseten.co/environments/production/sync/v1"
)

response = client.chat.completions.create(
    model="not-required",
    messages=[{"role": "user", "content": "Explain mixture of experts in two sentences."}],
)

Structured outputs and function calling are supported through the standard OpenAI parameters and have their own reference pages.

Observability

BIS-LLM emits metrics from three components. Each has its own dashboard section:

Domain	Metric prefix	Page
Autoscaler decisions	`autoscaler_*`	Autoscaling engines
Router and KV cache	`kv_cache_*`	KV-aware routing
Engine and request	engine-level metrics below	This page

Engine-level metrics, available on every BIS-LLM deployment:

Metric	What it measures
`tps_per_request`	Output token count divided by full request processing time, including time to first token. One sample per request that generates output tokens.
`decode_tokens_per_second`	Per-stream output token generation rate after the first token. One sample per streamed output chunk.
`input_tokens` / `output_tokens`	Total token throughput across the deployment.
`input_tokens_per_request` / `output_tokens_per_request`	Per-request token averages.
`concurrent_requests`	Currently in-flight request count.
`speculation_rate`	Draft-token acceptance rate when speculative decoding is active. High rates indicate the draft model is well-aligned.
`cpu_usage` / `memory_usage` / `gpu_usage` / `gpu_memory_usage`	Resource utilization per replica.
`replica_count_by_status`	Replica counts grouped by lifecycle status.

For more information about how tps_per_request, decode_tokens_per_second, and total output throughput differ, see LLM token speed. If you run Enterprise features, add kv_cache_hit_rate (KV-aware routing, in the router domain) or speculation_rate (Eagle/MTP) next. For more information about the configuration that produces speculation_rate, see Speculative decoding.

Migrate from Engine-Builder-LLM

The Engine-Builder trt_llm build schema has a v2 (inference_stack: v2) that moves runtime fields out of build:, renames tensor_parallel_count to tensor_parallel_size, and drops fields v2 handles automatically (plugin_configuration, base_model). For more information, see Migrate from Engine-Builder-LLM.

BIS-LLM configuration reference: Complete v2 configuration options.
Migrate from Engine-Builder-LLM: Translate a v1 configuration to BIS-LLM.
Advanced features for BIS-LLM: KV-aware routing, speculative decoding, and limited-availability disaggregated serving.
Autoscaling engines: Configure target in-flight tokens for BIS-LLM deployments.
Structured outputs: JSON schema validation.
Examples section: Concrete deployment examples.

Get started

Model APIs

Inference

Training

Development

Deployment

Engines

Model Labs

Organization

Hosting options

Observability

Troubleshooting

BIS-LLM features

Token-based autoscaling

KV-aware routing

Speculative decoding

A canonical configuration

OpenAI-compatible inference

Observability

Migrate from Engine-Builder-LLM

​BIS-LLM features

Token-based autoscaling

KV-aware routing

Speculative decoding

​A canonical configuration

​OpenAI-compatible inference

​Observability

​Migrate from Engine-Builder-LLM

​Related

BIS-LLM features

A canonical configuration

OpenAI-compatible inference

Observability

Migrate from Engine-Builder-LLM

Related