BIS-LLM features
BIS-LLM adds token-based autoscaling, KV-aware routing, and speculative decoding to the standard inference path. For more information, see Autoscaling engines and Advanced features for BIS-LLM.Token-based autoscaling
Scales replicas on
target_in_flight_tokens rather than request concurrency, so mixed-length prompt workloads scale on real compute load.KV-aware routing
Routes requests to the worker most likely to serve them from KV cache. Lower time-to-first-token on prefix-overlapping traffic.
Speculative decoding
Eagle, MTP, and N-gram speculation. Multiple tokens per forward pass on supported architectures.
A canonical configuration
BIS-LLM is configured in thebis_llm block of config.yaml. A prequantized DeepSeek V3.2 deployment on H200 looks like:
config.yaml
baseten model push, the BDN mirrors weights to GPU-local storage and the deployment exposes OpenAI-compatible /v1/chat/completions and /v1/responses, plus Anthropic-compatible /v1/messages. For more information, see BIS-LLM configuration.
For tuning advice on a specific or fine-tuned model, contact us.
OpenAI-compatible inference
BIS-LLM deployments expose OpenAI-compatible/v1/chat/completions, /v1/completions, /v1/responses, and /v1/embeddings (where applicable), plus Anthropic-compatible /v1/messages. OpenAI and Anthropic client SDKs can call the matching routes without modification. Set model to the model_name from your bis_llm config. The deployment returns 404 for a model name it doesn’t serve:
Responses API
Call the same deployment with the OpenAI Responses API, for example from the OpenAI Agents SDK.input on every call. For streaming, function tools, structured output, and unsupported fields, see Responses.
Structured outputs and function calling are supported through the standard OpenAI parameters and have their own reference pages.
Observability
BIS-LLM emits metrics from three components. Each has its own dashboard section:
Engine-level metrics, available on every BIS-LLM deployment:
For more information about how
tps_per_request, decode_tokens_per_second, and total output throughput differ, see LLM token speed. If you run Enterprise features, add kv_cache_hit_rate (KV-aware routing, in the router domain) or speculation_rate (Eagle/MTP) next. For more information about the configuration that produces speculation_rate, see Speculative decoding.
Migrate from Engine-Builder-LLM
The Engine-Buildertrt_llm build schema has a v2 (inference_stack: v2) that moves runtime fields out of build:, renames tensor_parallel_count to tensor_parallel_size, and drops fields v2 handles automatically (plugin_configuration, base_model). For more information, see Migrate from Engine-Builder-LLM.
Related
- BIS-LLM configuration reference: Complete v2 configuration options.
- Migrate from Engine-Builder-LLM: Translate a v1 configuration to BIS-LLM.
- Advanced features for BIS-LLM: KV-aware routing, speculative decoding, and limited-availability disaggregated serving.
- Autoscaling engines: Configure target in-flight tokens for BIS-LLM deployments.
- Structured outputs: JSON schema validation.
- Examples section: Concrete deployment examples.