> ## Documentation Index > Fetch the complete documentation index at: https://docs.baseten.co/llms.txt > Use this file to discover all available pages before exploring further. # Overview > Token-based autoscaling, KV-aware routing, and speculative decoding for MoE and large dense models export const BisLlmEnterpriseGate = ({feature, plural = false}) => { if (!feature) { return

Some BIS-LLM features are available on Enterprise plans only.{" "} Contact us{" "} to enable BIS-LLM or its Enterprise features.

; } return

{feature} {plural ? "are Enterprise features on BIS-LLM" : "is an Enterprise feature on BIS-LLM"}.{" "} Contact us{" "} to enable {plural ? "them" : "it"} for your deployment.

; }; BIS-LLM (Baseten Inference Stack v2) is the engine for Mixture of Experts (MoE) models and large dense LLMs. It targets MoE families (DeepSeek V3.x, Qwen3MoE, Kimi-K2, Llama 4, GLM-4.7, GPT-OSS 120B) and the largest dense models, where the standard request-based autoscaler and a single-server inference engine both leave performance on the table. The v2 stack adds token-based autoscaling, KV-aware routing, speculative decoding, expert parallel load balancing, and DP attention. Deployments mirror build artifacts to the [Baseten Delivery Network](/development/model/bdn) so cold starts stay fast. ## BIS-LLM features BIS-LLM adds token-based autoscaling, KV-aware routing, and speculative decoding to the standard inference path. For more information, see [Autoscaling engines](/engines/performance-concepts/autoscaling-engines#bis-llm) and [Advanced features for BIS-LLM](/engines/bis-llm/advanced-features). Scales replicas on `target_in_flight_tokens` rather than request concurrency, so mixed-length prompt workloads scale on real compute load. Routes requests to the worker most likely to serve them from KV cache. Lower time-to-first-token on prefix-overlapping traffic. Eagle, MTP, and N-gram speculation. Multiple tokens per forward pass on supported architectures. Disaggregated serving is a limited-availability option for selected workloads, not the default BIS-LLM topology. For more information, see [Disaggregated serving](/engines/bis-llm/advanced-features#disaggregated-serving). ## A canonical configuration BIS-LLM is configured in the `bis_llm` block of `config.yaml`. A prequantized DeepSeek V3.2 deployment on H200 looks like: ```yaml config.yaml theme={"system"} model_name: deepseek-v3-2 resources: accelerator: H200:4 use_gpu: true bis_llm: config: engine_backend: vllm checkpoint_name: nvidia/DeepSeek-V3.2-NVFP4 model_name: deepseek-ai/DeepSeek-V3.2 tensor_parallel_size: 4 engine_config: max_num_seqs: 64 max_num_batched_tokens: 16384 max_model_len: 20480 gpu_memory_utilization: 0.92 enable_prefix_caching: true enable_chunked_prefill: true dtype: auto trust_remote_code: true additional_autoscaling_config: metrics: - name: in_flight_tokens target: 30000 ``` After `baseten model push`, the BDN mirrors weights to GPU-local storage and the deployment exposes OpenAI-compatible `/v1/chat/completions`. For more information, see [BIS-LLM configuration](/engines/bis-llm/bis-llm-config). For tuning advice on a specific or fine-tuned model, [contact us](mailto:support@baseten.co). ## OpenAI-compatible inference BIS-LLM deployments expose `/v1/chat/completions`, `/v1/completions`, and `/v1/embeddings` (where applicable). Standard OpenAI client SDKs work without modification: ```python theme={"system"} from openai import OpenAI import os client = OpenAI( api_key=os.environ["BASETEN_API_KEY"], base_url="https://model-xxxxxx.api.baseten.co/environments/production/sync/v1" ) response = client.chat.completions.create( model="not-required", messages=[{"role": "user", "content": "Explain mixture of experts in two sentences."}], ) ``` [Structured outputs](/inference/structured-outputs) and [function calling](/inference/function-calling) are supported through the standard OpenAI parameters and have their own reference pages. ## Observability BIS-LLM emits metrics from three components. Each has its own dashboard section: | Domain | Metric prefix | Page | | -------------------- | -------------------------- | ----------------------------------------------------------------------------------- | | Autoscaler decisions | `autoscaler_*` | [Autoscaling engines](/engines/performance-concepts/autoscaling-engines#monitoring) | | Router and KV cache | `kv_cache_*` | [KV-aware routing](/engines/bis-llm/advanced-features#kv-aware-routing) | | Engine and request | engine-level metrics below | This page | Engine-level metrics, available on every BIS-LLM deployment: | Metric | What it measures | | --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `tps_per_request` | Output token count divided by full request processing time, including time to first token. One sample per request that generates output tokens. | | `decode_tokens_per_second` | Per-stream output token generation rate after the first token. One sample per streamed output chunk. | | `input_tokens` / `output_tokens` | Total token throughput across the deployment. | | `input_tokens_per_request` / `output_tokens_per_request` | Per-request token averages. | | `concurrent_requests` | Currently in-flight request count. | | `speculation_rate` | Draft-token acceptance rate when speculative decoding is active. High rates indicate the draft model is well-aligned. | | `cpu_usage` / `memory_usage` / `gpu_usage` / `gpu_memory_usage` | Resource utilization per replica. | | `replica_count_by_status` | Replica counts grouped by lifecycle status. | For more information about how `tps_per_request`, `decode_tokens_per_second`, and total output throughput differ, see [LLM token speed](/observability/metrics#llm-token-speed). If you run Enterprise features, add `kv_cache_hit_rate` (KV-aware routing, in the router domain) or `speculation_rate` (Eagle/MTP) next. For more information about the configuration that produces `speculation_rate`, see [Speculative decoding](/engines/bis-llm/advanced-features#speculative-decoding). ## Migrate from Engine-Builder-LLM The Engine-Builder `trt_llm` build schema has a v2 (`inference_stack: v2`) that moves runtime fields out of `build:`, renames `tensor_parallel_count` to `tensor_parallel_size`, and drops fields v2 handles automatically (`plugin_configuration`, `base_model`). For more information, see [Migrate from Engine-Builder-LLM](/engines/bis-llm/migrate-from-v1). ## Related * [BIS-LLM configuration reference](/engines/bis-llm/bis-llm-config): Complete v2 configuration options. * [Migrate from Engine-Builder-LLM](/engines/bis-llm/migrate-from-v1): Translate a v1 configuration to BIS-LLM. * [Advanced features for BIS-LLM](/engines/bis-llm/advanced-features): KV-aware routing, speculative decoding, and limited-availability disaggregated serving. * [Autoscaling engines](/engines/performance-concepts/autoscaling-engines#bis-llm): Configure target in-flight tokens for BIS-LLM deployments. * [Structured outputs](/inference/structured-outputs): JSON schema validation. * [Examples section](/examples/overview): Concrete deployment examples.