Performance optimization

• Latency: How quickly does each user get output from the model?
• Throughput: How many requests can the deployment handle at once?
• Cost: How much does a standardized unit of work cost?
• Quality: Does your model consistently deliver high-quality output after optimization?

​

Performance engines

​

Engine-Builder-LLM: dense models

• Best for: Llama, Mistral, Qwen, and other causal language models.
• Features: TensorRT-LLM optimization, lookahead decoding, quantization.
• Performance: Tuned for low-latency, high-throughput dense LLM inference.

​

BIS-LLM: MoE models

• Best for: DeepSeek, Mixtral, and other mixture-of-experts models.
• Features: V2 inference stack, expert routing, structured outputs.
• Performance: Tuned for large-scale MoE inference.

​

BEI: embedding models

• Best for: Sentence transformers, rerankers, classification models.
• Features: OpenAI-compatible API, optimized batching.
• Performance: Tuned for high-throughput embedding inference.

​

Performance concepts

• Quantization guide: FP8 and FP4 trade-offs and hardware requirements.
• Structured outputs: JSON schema validation and controlled generation.
• Function calling: tool use and function selection.
• Performance client: high-throughput client library.
• Deploy from cloud storage: GCS, S3, and Azure with Engine-Builder-LLM.
• Deploy with inference engines: Baseten Training checkpoints with TRT-LLM.

​

Quick performance wins

​

Quantization

trt_llm:
  build:
    quantization_type: fp8  # FP8 weights, 16-bit KV cache

​

Lookahead decoding

trt_llm:
  build:
    speculator:
      speculative_decoding_mode: LOOKAHEAD_DECODING
      lookahead_windows_size: 3

​

Performance client

uv pip install baseten-performance-client

​

Where to start

1. Choose your engine: Engine selection
2. Configure your model: Engine-specific configuration guides
3. Optimize performance: Performance concepts
4. Deploy and monitor: Use performance client for maximum throughput