To install without Homebrew, see the Baseten CLI install reference.Pick the model you want to deploy. Each tab is a self-contained recipe.
Mini 4B
Mini 3B Batch
mistralai/Voxtral-Mini-4B-Realtime-2602 is a 4B-parameter encoder-decoder model.This preset serves Voxtral Mini Realtime on an RTX PRO 6000, tuned for low-latency streaming transcription.
Hardware
RTX-PRO-6000
Engine
vLLM (0.22.0 custom build)
Context
10K
Concurrency
48
Write the config
Create and move into the project directory:
mkdir voxtral-mini-4b-streaming && cd voxtral-mini-4b-streaming
Then create a file named config.yaml and paste the following:
config.yaml
model_name: "model:voxtral-mini-4b preset:latency"model_metadata: repo_id: mistralai/Voxtral-Mini-4B-Realtime-2602weights: - source: "hf://mistralai/Voxtral-Mini-4B-Realtime-2602@2769294da9567371363522aac9bbcfdd19447add" mount_location: "/app/checkpoint/model"# VLLM_CACHE_ROOT, TORCHINDUCTOR_CACHE_DIR and TRITON_CACHE_DIR stay unset: the platform's# startup cache sets them to paths it restores into every new pod. Pointing them at the shared# /cache/org mount (as this preset did) skips that restore, and on another preset it hung startup# until the 3000 s check killed it (#1077, #1079); tests/test_registry_jit_cache_dirs.py.environment_variables: {}base_image: image: "baseten/vllm-openai:0.22.0-voxtral-realtime-fixes"docker_server: start_command: >- sh -c "vllm serve /app/checkpoint/model --tensor-parallel-size 1 --api-server-count 8 --enable-realtime-unbounded --no-enable-prefix-caching --realtime-reanchor-margin-tokens 1024 --hf-overrides '{\"text_config\": {\"sliding_window\": 4096}}' --served-model-name mistralai/Voxtral-Mini-4B-Realtime-2602 --host 0.0.0.0 --port 8000 --max-num-seqs 48 --max-model-len 10240 --compilation-config '{\"cudagraph_mode\": \"PIECEWISE\", \"cudagraph_capture_sizes\": [1, 2, 4, 8, 16, 24, 32, 48], \"max_cudagraph_capture_size\": 48}'" readiness_endpoint: /health liveness_endpoint: /health predict_endpoint: /v1/realtime server_port: 8000resources: instance_type: RTX-PRO-6000requirements: - librosa - pynvml - ffmpeg-python - websocketssystem_packages: # No python3.10-venv here: nothing in this truss builds a venv (vLLM is started # via docker_server), and pinning a Python minor version in apt ties the build to # whatever the base image happens to ship. That pin already broke the community-1 # diarizer when its base moved to Debian trixie: # E: Unable to locate package python3.10-venv - ffmpeg - openmpi-bin - libopenmpi-devruntime: predict_concurrency: 48 is_websocket_endpoint: true transport: kind: websocket ping_interval_seconds: null ping_timeout_seconds: null
Flags
The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:
Flag
Value
What it does
--tensor-parallel-size
1
Number of GPUs to shard the model across.
--api-server-count
8
Number of API server processes vLLM runs in front of the engine, spreading HTTP and WebSocket handling across CPU cores.
--enable-realtime-unbounded
(no value)
Lets realtime streaming sessions run without a fixed duration cap. Available in Baseten’s patched vLLM build, not upstream vLLM.
--no-enable-prefix-caching
(no value)
Disable prefix caching, so repeated prompts do not reuse cached KV blocks.
--realtime-reanchor-margin-tokens
1024
Token margin the realtime endpoint keeps when it re-anchors a long stream’s context window, bounding memory growth over long sessions.
--hf-overrides
{"text_config": {"sliding_window": 4096}}
Overrides fields in the model’s Hugging Face config as a JSON object. The dotted form (--hf-overrides.<field>) sets the same fields one at a time.
--max-num-seqs
48
Maximum number of concurrent sequences in the batch.
--max-model-len
10240
Maximum context length (tokens) the server accepts per request.
Pushing model "voxtral-mini-4b-streaming"...Uploading model...Uploaded model in 0s✨ Model voxtral-mini-4b-streaming was successfully pushed ✨ Model: voxtral-mini-4b-streaming (abc1d2ef) Deployment: xyz123 Environment: production🪵 View logs: deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123 environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed) app: https://app.baseten.co/models/abc1d2ef/logs/xyz123🚀 Invoke your model: URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict CLI: baseten model predict --model-id abc1d2ef
baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.
Call the model
This preset exposes a WebSocket streaming endpoint at /v1/realtime for low-latency, incremental transcription. See the streaming transcription API reference for the message protocol, Python client example, and supported audio formats.
mistralai/Voxtral-Mini-3B-2507 is a 4.7B-parameter encoder-decoder model with up to 32K context.This preset serves Voxtral Mini 3B on H100 40GB, tuned for batch transcription of pre-recorded audio.
Hardware
H100MIG
Engine
vLLM (0.22.0-cu129 build)
Context
32K
Concurrency
256
Write the config
Create and move into the project directory:
mkdir voxtral-mini-3b-batch && cd voxtral-mini-3b-batch
Then create a file named config.yaml and paste the following:
config.yaml
model_name: "model:voxtral-mini-3b preset:latency"model_metadata: repo_id: mistralai/Voxtral-Mini-3B-2507 tags: - openai-compatible - audio - asrweights: - source: "hf://mistralai/Voxtral-Mini-3B-2507@3060fe34b35ba5d44202ce9ff3c097642914f8f3" mount_location: "/app/checkpoint/model" # The HF repo ships the weights twice: mistral-format # consolidated.safetensors (what --load-format mistral reads) and an # HF-format shard pair + model.safetensors.index.json. Skip the unused # HF shards so the mount doesn't stream ~9GB of duplicate weights. ignore_patterns: - "model-*.safetensors" - "model.safetensors.index.json"base_image: image: vllm/vllm-openai:v0.22.0-cu129docker_server: # Voxtral ships mistral-format artifacts only (params.json + tekken.json, # no HF tokenizer files), so vLLM needs the mistral tokenizer/config/load # trio per the upstream serving recipe. start_command: >- sh -c "vllm serve /app/checkpoint/model --tokenizer-mode mistral --config-format mistral --load-format mistral --tensor-parallel-size 1 --served-model-name mistralai/Voxtral-Mini-3B-2507 --max-model-len 32768 --gpu-memory-utilization 0.8 --host 0.0.0.0 --port 8000" readiness_endpoint: /health liveness_endpoint: /health predict_endpoint: /v1/audio/transcriptions server_port: 8000resources: instance_type: H100MIGrequirements: - vllm[audio] - mistral-common[audio] - librosa - torch - torchaudio - pynvml - ffmpeg-pythonsystem_packages: # No python3.10-venv here: nothing in this truss builds a venv (vLLM is started # via docker_server), and pinning a Python minor version in apt ties the build to # whatever the base image happens to ship. That pin already broke the community-1 # diarizer when its base moved to Debian trixie: # E: Unable to locate package python3.10-venv - ffmpeg - openmpi-bin - libopenmpi-devruntime: predict_concurrency: 256
Your deployment exposes the OpenAI-compatible /v1/audio/transcriptions endpoint, so the OpenAI Python SDK works with only a base_url change. Voxtral Mini 3B detects the spoken language automatically across 8 supported languages and handles audio up to about 30 minutes per request.
Flags
The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:
Flag
Value
What it does
--tokenizer-mode
mistral
Selects a custom tokenizer implementation. Required for models that ship a non-standard tokenizer alongside the checkpoint.
--config-format
mistral
Model config format. Use mistral for models that ship Mistral-format params.json configs instead of HF config.json.
--load-format
mistral
Weight loading backend.
--tensor-parallel-size
1
Number of GPUs to shard the model across.
--max-model-len
32768
Maximum context length (tokens) the server accepts per request.
--gpu-memory-utilization
0.8
Fraction of GPU memory vLLM may use for weights and KV cache.
Deploy
Deploy the model with the Baseten CLI:
Baseten CLI
baseten model push
You should see output similar to:
Pushing model "voxtral-mini-3b-batch"...Uploading model...Uploaded model in 0s✨ Model voxtral-mini-3b-batch was successfully pushed ✨ Model: voxtral-mini-3b-batch (abc1d2ef) Deployment: xyz123 Environment: production🪵 View logs: deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123 environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed) app: https://app.baseten.co/models/abc1d2ef/logs/xyz123🚀 Invoke your model: URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict CLI: baseten model predict --model-id abc1d2ef
baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.
Call the model
Your deployment serves the OpenAI-compatible transcription endpoint at /v1/audio/transcriptions.Upload an audio file as multipart form data. The endpoint returns the transcription in the response body.