Model performance means optimizing every layer of your model serving infrastructure to balance four goals:
- Latency: How quickly does each user get output from the model?
- Throughput: How many requests can the deployment handle at once?
- Cost: How much does a standardized unit of work cost?
- Quality: Does your model consistently deliver high-quality output after optimization?
Baseten provides three managed inference engines. Pick the one that matches your model architecture:
- Best for: Llama, Mistral, Qwen, and other causal language models.
- Features: TensorRT-LLM optimization, lookahead decoding, quantization.
- Performance: Tuned for low-latency, high-throughput dense LLM inference.
- Best for: DeepSeek, Mixtral, and other mixture-of-experts models.
- Features: V2 inference stack, expert routing, structured outputs.
- Performance: Tuned for large-scale MoE inference.
BEI: embedding models
- Best for: Sentence transformers, rerankers, classification models.
- Features: OpenAI-compatible API, optimized batching.
- Performance: Tuned for high-throughput embedding inference.
Detailed optimization guides live in the performance concepts section:
Quantization
Reduce weight memory and improve throughput with post-training quantization:
trt_llm:
build:
quantization_type: fp8 # FP8 weights, 16-bit KV cache
fp8, fp8_kv, fp4, fp4_kv, fp4_mlp_only).
Lookahead decoding
Accelerate inference for predictable content like code or JSON:
trt_llm:
build:
speculator:
speculative_decoding_mode: LOOKAHEAD_DECODING
lookahead_windows_size: 3
uv pip install baseten-performance-client
Where to start
- Choose your engine: Engine selection
- Configure your model: Engine-specific configuration guides
- Optimize performance: Performance concepts
- Deploy and monitor: Use performance client for maximum throughput
Start with the default engine configuration, then apply quantization and other optimizations based on your specific performance requirements.
