> ## Documentation Index > Fetch the complete documentation index at: https://docs.baseten.co/llms.txt > Use this file to discover all available pages before exploring further. # Voxtral > Voxtral recipes: 2 variants (Mini 4B, Mini 3B Batch), encoder-decoder architecture.
Speech-to-text
## Setup Install the Baseten CLI and sign in, then install the `websockets` library. **Install and sign in to Baseten** ```bash Terminal theme={"system"} brew tap basetenlabs/baseten brew trust --formula basetenlabs/baseten/baseten brew install baseten ``` Download and extract the Windows archive for the latest Baseten CLI release: ```powershell Terminal theme={"system"} $release = Invoke-RestMethod https://api.github.com/repos/basetenlabs/baseten-cli/releases/latest $asset = $release.assets | Where-Object { $_.name -like "*_windows_amd64.zip" } Invoke-WebRequest $asset.browser_download_url -OutFile baseten.zip Expand-Archive -Force baseten.zip . ``` Move `baseten.exe` to a directory on your `PATH`. For other platforms or a specific version, see the [Baseten CLI install reference](/reference/cli/baseten/overview#install). ```sh theme={"system"} baseten auth login ``` **Install websockets** ```sh theme={"system"} uv pip install websockets ``` To install without Homebrew, see the [Baseten CLI install reference](/reference/cli/baseten/overview#install). Pick the model you want to deploy. Each tab is a self-contained recipe. [mistralai/Voxtral-Mini-4B-Realtime-2602](https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602) is a 4B-parameter encoder-decoder model. This preset serves Voxtral Mini Realtime on an RTX PRO 6000, tuned for low-latency streaming transcription. RTX-PRO-6000 vLLM (0.22.0 custom build) 10K 48 ## Write the config Create and move into the project directory: ```sh theme={"system"} mkdir voxtral-mini-4b-streaming && cd voxtral-mini-4b-streaming ``` Then create a file named `config.yaml` and paste the following: ```yaml config.yaml theme={"system"} model_name: "model:voxtral-mini-4b preset:latency" model_metadata: repo_id: mistralai/Voxtral-Mini-4B-Realtime-2602 weights: - source: "hf://mistralai/Voxtral-Mini-4B-Realtime-2602@2769294da9567371363522aac9bbcfdd19447add" mount_location: "/app/checkpoint/model" environment_variables: VLLM_CACHE_ROOT: /cache/org/vllm TORCHINDUCTOR_CACHE_DIR: /cache/org/inductor TRITON_CACHE_DIR: /cache/org/triton base_image: image: "baseten/vllm-openai:0.22.0-voxtral-realtime-fixes" docker_server: start_command: >- sh -c "vllm serve /app/checkpoint/model --tensor-parallel-size 1 --api-server-count 8 --enable-realtime-unbounded --no-enable-prefix-caching --realtime-reanchor-margin-tokens 1024 --hf-overrides '{\"text_config\": {\"sliding_window\": 4096}}' --served-model-name mistralai/Voxtral-Mini-4B-Realtime-2602 --host 0.0.0.0 --port 8000 --max-num-seqs 48 --max-model-len 10240 --compilation-config '{\"cudagraph_mode\": \"PIECEWISE\", \"cudagraph_capture_sizes\": [1, 2, 4, 8, 16, 24, 32, 48], \"max_cudagraph_capture_size\": 48}'" readiness_endpoint: /health liveness_endpoint: /health predict_endpoint: /v1/realtime server_port: 8000 resources: instance_type: RTX-PRO-6000 requirements: - librosa - pynvml - ffmpeg-python - websockets system_packages: # No python3.10-venv here: nothing in this truss builds a venv (vLLM is started # via docker_server), and pinning a Python minor version in apt ties the build to # whatever the base image happens to ship. That pin already broke the community-1 # diarizer when its base moved to Debian trixie: # E: Unable to locate package python3.10-venv - ffmpeg - openmpi-bin - libopenmpi-dev runtime: predict_concurrency: 48 is_websocket_endpoint: true transport: kind: websocket ping_interval_seconds: null ping_timeout_seconds: null ``` ## Flags The `start_command` passes these flags to the engine. Each one controls a runtime or serving behavior: | Flag | Value | What it does | | - | - | - | | `--tensor-parallel-size` | `1` | Number of GPUs to shard the model across. | | `--api-server-count` | `8` | Number of API server processes vLLM runs in front of the engine, spreading HTTP and WebSocket handling across CPU cores. | | `--enable-realtime-unbounded` | (no value) | Lets realtime streaming sessions run without a fixed duration cap. Available in Baseten's patched vLLM build, not upstream vLLM. | | `--no-enable-prefix-caching` | (no value) | Disable prefix caching, so repeated prompts do not reuse cached KV blocks. | | `--realtime-reanchor-margin-tokens` | `1024` | Token margin the realtime endpoint keeps when it re-anchors a long stream's context window, bounding memory growth over long sessions. | | `--hf-overrides` | `{"text_config": {"sliding_window": 4096}}` | Overrides fields in the model's Hugging Face config as a JSON object. The dotted form (`--hf-overrides.`) sets the same fields one at a time. | | `--max-num-seqs` | `48` | Maximum number of concurrent sequences in the batch. | | `--max-model-len` | `10240` | Maximum context length (tokens) the server accepts per request. | | `--compilation-config` | `{"cudagraph_mode": "PIECEWISE", "cudagraph_capture_sizes": [1, 2, 4, 8, 16, 24, 32, 48], "max_cudagraph_capture_size": 48}` | vLLM compilation passes (op fusion, dead-code elimination). | ## Deploy Deploy the model with the Baseten CLI: ```sh Baseten CLI theme={"system"} baseten model push ``` You should see output similar to: ```output theme={"system"} Pushing model "voxtral-mini-4b-streaming"... Uploading model... Uploaded model in 0s ✨ Model voxtral-mini-4b-streaming was successfully pushed ✨ Model: voxtral-mini-4b-streaming (abc1d2ef) Deployment: xyz123 Environment: production 🪵 View logs: deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123 environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed) app: https://app.baseten.co/models/abc1d2ef/logs/xyz123 🚀 Invoke your model: URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict CLI: baseten model predict --model-id abc1d2ef ``` `baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable. ## Call the model This preset exposes a WebSocket streaming endpoint at `/v1/realtime` for low-latency, incremental transcription. See the [streaming transcription API reference](/reference/inference-api/predict-endpoints/streaming-transcription-api) for the message protocol, Python client example, and supported audio formats. [mistralai/Voxtral-Mini-3B-2507](https://huggingface.co/mistralai/Voxtral-Mini-3B-2507) is a 4.7B-parameter encoder-decoder model with up to 32K context. This preset serves Voxtral Mini 3B on H100 40GB, tuned for batch transcription of pre-recorded audio. H100MIG vLLM (0.22.0-cu129 build) 32K 256 ## Write the config Create and move into the project directory: ```sh theme={"system"} mkdir voxtral-mini-3b-batch && cd voxtral-mini-3b-batch ``` Then create a file named `config.yaml` and paste the following: ```yaml config.yaml theme={"system"} model_name: "model:voxtral-mini-3b preset:latency" model_metadata: repo_id: mistralai/Voxtral-Mini-3B-2507 tags: - openai-compatible - audio - asr weights: - source: "hf://mistralai/Voxtral-Mini-3B-2507@3060fe34b35ba5d44202ce9ff3c097642914f8f3" mount_location: "/app/checkpoint/model" # The HF repo ships the weights twice: mistral-format # consolidated.safetensors (what --load-format mistral reads) and an # HF-format shard pair + model.safetensors.index.json. Skip the unused # HF shards so the mount doesn't stream ~9GB of duplicate weights. ignore_patterns: - "model-*.safetensors" - "model.safetensors.index.json" base_image: image: vllm/vllm-openai:v0.22.0-cu129 docker_server: # Voxtral ships mistral-format artifacts only (params.json + tekken.json, # no HF tokenizer files), so vLLM needs the mistral tokenizer/config/load # trio per the upstream serving recipe. start_command: >- sh -c "vllm serve /app/checkpoint/model --tokenizer-mode mistral --config-format mistral --load-format mistral --tensor-parallel-size 1 --served-model-name mistralai/Voxtral-Mini-3B-2507 --max-model-len 32768 --gpu-memory-utilization 0.8 --host 0.0.0.0 --port 8000" readiness_endpoint: /health liveness_endpoint: /health predict_endpoint: /v1/audio/transcriptions server_port: 8000 resources: instance_type: H100MIG requirements: - vllm[audio] - mistral-common[audio] - librosa - torch - torchaudio - pynvml - ffmpeg-python system_packages: # No python3.10-venv here: nothing in this truss builds a venv (vLLM is started # via docker_server), and pinning a Python minor version in apt ties the build to # whatever the base image happens to ship. That pin already broke the community-1 # diarizer when its base moved to Debian trixie: # E: Unable to locate package python3.10-venv - ffmpeg - openmpi-bin - libopenmpi-dev runtime: predict_concurrency: 256 ``` Your deployment exposes the OpenAI-compatible `/v1/audio/transcriptions` endpoint, so the OpenAI Python SDK works with only a `base_url` change. Voxtral Mini 3B detects the spoken language automatically across 8 supported languages and handles audio up to about 30 minutes per request. ## Flags The `start_command` passes these flags to the engine. Each one controls a runtime or serving behavior: | Flag | Value | What it does | | - | - | - | | `--tokenizer-mode` | `mistral` | Selects a custom tokenizer implementation. Required for models that ship a non-standard tokenizer alongside the checkpoint. | | `--config-format` | `mistral` | Model config format. Use `mistral` for models that ship Mistral-format `params.json` configs instead of HF `config.json`. | | `--load-format` | `mistral` | Weight loading backend. | | `--tensor-parallel-size` | `1` | Number of GPUs to shard the model across. | | `--max-model-len` | `32768` | Maximum context length (tokens) the server accepts per request. | | `--gpu-memory-utilization` | `0.8` | Fraction of GPU memory vLLM may use for weights and KV cache. | ## Deploy Deploy the model with the Baseten CLI: ```sh Baseten CLI theme={"system"} baseten model push ``` You should see output similar to: ```output theme={"system"} Pushing model "voxtral-mini-3b-batch"... Uploading model... Uploaded model in 0s ✨ Model voxtral-mini-3b-batch was successfully pushed ✨ Model: voxtral-mini-3b-batch (abc1d2ef) Deployment: xyz123 Environment: production 🪵 View logs: deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123 environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed) app: https://app.baseten.co/models/abc1d2ef/logs/xyz123 🚀 Invoke your model: URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict CLI: baseten model predict --model-id abc1d2ef ``` `baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable. ## Call the model Your deployment serves the OpenAI-compatible transcription endpoint at `/v1/audio/transcriptions`. Upload an audio file as multipart form data. The endpoint returns the transcription in the response body. ```python main.py theme={"system"} import os from openai import OpenAI client = OpenAI( api_key=os.environ["BASETEN_API_KEY"], base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1", ) with open("audio.wav", "rb") as audio_file: response = client.audio.transcriptions.create( model="mistralai/Voxtral-Mini-3B-2507", file=audio_file, ) print(response.text) ``` ```sh theme={"system"} curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/audio/transcriptions \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -F "file=@audio.wav" \ -F "model=mistralai/Voxtral-Mini-3B-2507" ``` ## Next steps Endpoint anatomy, authentication, and sync versus async inference Scale replicas with traffic, including scale to zero