> ## Documentation Index > Fetch the complete documentation index at: https://docs.baseten.co/llms.txt > Use this file to discover all available pages before exploring further. # VibeVoice > Microsoft's VibeVoice-ASR speech-to-text model, returning JSON segments with speaker labels and timestamps through an OpenAI-compatible API.
Speech-to-text
## Setup Sign in to Baseten with Truss, then install the OpenAI SDK. **Sign in to Baseten** ```sh theme={"system"} uvx truss login --browser ``` **Install the OpenAI SDK** ```sh theme={"system"} uv pip install openai ``` [microsoft/VibeVoice-ASR](https://huggingface.co/microsoft/VibeVoice-ASR) is a multimodal speech-to-text model. This preset serves VibeVoice-ASR on a single H100 through vLLM with an OpenAI-compatible chat completions endpoint, tuned for low-latency transcription with speaker labels and timestamps. H100 vLLM 0.14.1 32K 32 ## Write the config Create and move into the project directory: ```sh theme={"system"} mkdir vibevoice-asr-latency && cd vibevoice-asr-latency ``` Then create a file named `config.yaml` and paste the following: ```yaml config.yaml theme={"system"} model_name: "model:vibevoice-asr preset:latency" python_version: py310 model_metadata: repo_id: microsoft/VibeVoice-ASR tags: - openai-compatible - audio - asr - speech-to-text example_model_input: model: vibevoice messages: - role: system content: You are a helpful assistant that transcribes audio input into text output in JSON format. - role: user content: - type: audio_url audio_url: url: https://github.com/ggerganov/whisper.cpp/raw/master/samples/jfk.wav - type: text text: Transcribe this audio. max_tokens: 64 temperature: 0.0 base_image: image: vllm/vllm-openai:v0.14.1 python_executable_path: /usr/bin/python3 environment_variables: HF_HOME: /cache/org HF_HUB_CACHE: /cache/org TRANSFORMERS_CACHE: /cache/org VIBEVOICE_FFMPEG_MAX_CONCURRENCY: "64" VLLM_MEDIA_LOADING_THREAD_COUNT: "16" PYTORCH_ALLOC_CONF: "expandable_segments:True" requirements: - transformers==4.57.6 - accelerate>=0.30.0 - safetensors - huggingface-hub>=0.23.0 - librosa>=0.10.0 - soundfile - scipy - pydub - diffusers - git+https://github.com/microsoft/VibeVoice.git@main resources: accelerator: H100 cpu: "4" memory: 32Gi use_gpu: true runtime: predict_concurrency: 32 secrets: hf_access_token: null system_packages: - ffmpeg - git # Weights are pre-downloaded at build time and mounted at /models/vibevoice-asr, # so cold starts skip the 9.2 GB HF download entirely. weights: - source: "hf://microsoft/VibeVoice-ASR@main" mount_location: "/models/vibevoice-asr" auth_secret_name: "hf_access_token" # Pass-through mode: no model.py, Truss just runs vllm serve and proxies # /predict requests to /v1/chat/completions on the container's localhost. docker_server: server_port: 8000 predict_endpoint: /v1/chat/completions readiness_endpoint: /v1/models liveness_endpoint: /v1/models start_command: | bash -c ' set -e echo "[entrypoint] Applying microsoft/VibeVoice plugin patches..." python3 /app/data/patch.py echo "[entrypoint] Generating tokenizer files..." python3 -m vllm_plugin.tools.generate_tokenizer_files --output /models/vibevoice-asr echo "[entrypoint] Starting vLLM serve..." exec vllm serve /models/vibevoice-asr \ --served-model-name vibevoice \ --trust-remote-code \ --dtype bfloat16 \ --max-num-seqs 16 \ --max-model-len 32768 \ --gpu-memory-utilization 0.85 \ --num-gpu-blocks-override 4096 \ --no-enable-prefix-caching \ --enable-chunked-prefill \ --chat-template-content-format openai \ --allowed-local-media-path /app \ --media-io-kwargs "{\"audio\": {\"target_sr\": 24000}}" \ --enforce-eager \ --skip-mm-profiling \ --host 0.0.0.0 \ --port 8000 ' ``` This config runs the `vllm/vllm-openai:v0.14.1` image with Microsoft's VibeVoice plugin patches applied at startup, serving weights pre-mounted at `/models/vibevoice-asr` so cold starts skip the 9.2 GB Hugging Face download. The server runs in eager mode with a 32k context and up to 16 concurrent sequences, exposing the model as `vibevoice` on the chat completions endpoint. ## Flags The `start_command` passes these flags to the engine. Each one controls a runtime or serving behavior: | Flag | Value | What it does | | -------------------------------- | --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | `--trust-remote-code` | (no value) | Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures). | | `--dtype` | `bfloat16` | Weight precision loaded at runtime. **bfloat16:** BF16 weights, no quantization. | | `--max-num-seqs` | `16` | Maximum number of concurrent sequences in the batch. | | `--max-model-len` | `32768` | Maximum context length (tokens) the server accepts per request. | | `--gpu-memory-utilization` | `0.85` | Fraction of GPU memory vLLM may use for weights and KV cache. | | `--num-gpu-blocks-override` | `4096` | Overrides vLLM's profiled KV cache size with a fixed number of GPU blocks. | | `--no-enable-prefix-caching` | (no value) | Disable prefix caching, so repeated prompts do not reuse cached KV blocks. | | `--enable-chunked-prefill` | (no value) | Process long prompts in chunks so decode requests keep running. | | `--chat-template-content-format` | `openai` | Format the chat template uses to render message content. **openai:** OpenAI-style content parts (list of typed segments) rather than a plain string. | | `--allowed-local-media-path` | `/app` | Filesystem path the server may read local media files from when resolving multimodal inputs. | | `--media-io-kwargs` | `{"audio": {"target_sr": 24000}}` | Options passed to the multimodal media loaders as a JSON object, for example the target sample rate for audio inputs. | | `--enforce-eager` | (no value) | Run the model in eager mode instead of capturing CUDA graphs. | | `--skip-mm-profiling` | (no value) | Skip multimodal memory profiling at startup, reducing startup time. | ## Deploy Push the config to Baseten: ```sh theme={"system"} uvx truss push ``` You should see output similar to: ```output theme={"system"} ✨ Model vibevoice-asr-latency was successfully pushed ✨ Model ID: abc1d2ef Deployment ID: xyz123 Endpoint: model-abc1d2ef.api.baseten.co Logs: https://app.baseten.co/models/abc1d2ef/logs/xyz123 ``` Your **model ID** is printed in the `truss push` output (`abcd1234` in the example). Use it wherever you see `{model_id}` in the next section. ## Call the model Your deployment serves an OpenAI-compatible chat completions API at `/v1/chat/completions` that accepts audio inputs. Replace `{model_id}` with your model ID and make sure `BASETEN_API_KEY` is set. Send audio as an `audio_url` content item on a chat message. The model returns the transcription as the assistant message content. ```python main.py theme={"system"} import os from openai import OpenAI client = OpenAI( api_key=os.environ["BASETEN_API_KEY"], base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1", ) response = client.chat.completions.create( model="vibevoice", messages=[ { "role": "user", "content": [ { "type": "audio_url", "audio_url": { "url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav" }, } ], } ], ) print(response.choices[0].message.content) ``` ```sh theme={"system"} curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{ "model": "vibevoice", "messages": [ {"role": "user", "content": [ {"type": "audio_url", "audio_url": {"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav"}} ]} ] }' ```