VibeVoice - Baseten

Setup

Sign in to Baseten

uvx truss login --browser

Install the OpenAI SDK

uv pip install openai

microsoft/VibeVoice-ASR is a multimodal speech-to-text model. This preset serves VibeVoice-ASR on a single H100 through vLLM with an OpenAI-compatible chat completions endpoint, tuned for low-latency transcription with speaker labels and timestamps.

Hardware

H100

Engine

vLLM 0.14.1

Context

32K

Concurrency

Write the config

Create and move into the project directory:

mkdir vibevoice-asr-latency && cd vibevoice-asr-latency

Then create a file named config.yaml and paste the following:

config.yaml

model_name: "model:vibevoice-asr preset:latency"
python_version: py310

model_metadata:
  repo_id: microsoft/VibeVoice-ASR
  tags:
    - openai-compatible
    - audio
    - asr
    - speech-to-text
  example_model_input:
    model: vibevoice
    messages:
      - role: system
        content: You are a helpful assistant that transcribes audio input into text output in JSON format.
      - role: user
        content:
          - type: audio_url
            audio_url:
              url: https://github.com/ggerganov/whisper.cpp/raw/master/samples/jfk.wav
          - type: text
            text: Transcribe this audio.
    max_tokens: 64
    temperature: 0.0

base_image:
  image: vllm/vllm-openai:v0.14.1
  python_executable_path: /usr/bin/python3

environment_variables:
  HF_HOME: /cache/org
  HF_HUB_CACHE: /cache/org
  TRANSFORMERS_CACHE: /cache/org
  VIBEVOICE_FFMPEG_MAX_CONCURRENCY: "64"
  VLLM_MEDIA_LOADING_THREAD_COUNT: "16"
  PYTORCH_ALLOC_CONF: "expandable_segments:True"

requirements:
  - transformers==4.57.6
  - accelerate>=0.30.0
  - safetensors
  - huggingface-hub>=0.23.0
  - librosa>=0.10.0
  - soundfile
  - scipy
  - pydub
  - diffusers
  - git+https://github.com/microsoft/VibeVoice.git@main

resources:
  accelerator: H100
  cpu: "4"
  memory: 32Gi
  use_gpu: true

runtime:
  predict_concurrency: 32

secrets:
  hf_access_token: null

system_packages:
  - ffmpeg
  - git

# Weights are pre-downloaded at build time and mounted at /models/vibevoice-asr,
# so cold starts skip the 9.2 GB HF download entirely.
weights:
  - source: "hf://microsoft/VibeVoice-ASR@main"
    mount_location: "/models/vibevoice-asr"
    auth_secret_name: "hf_access_token"

# Pass-through mode: no model.py, Truss just runs vllm serve and proxies
# /predict requests to /v1/chat/completions on the container's localhost.
docker_server:
  server_port: 8000
  predict_endpoint: /v1/chat/completions
  readiness_endpoint: /v1/models
  liveness_endpoint: /v1/models
  start_command: |
    bash -c '
    set -e
    echo "[entrypoint] Applying microsoft/VibeVoice plugin patches..."
    python3 /app/data/patch.py
    echo "[entrypoint] Generating tokenizer files..."
    python3 -m vllm_plugin.tools.generate_tokenizer_files --output /models/vibevoice-asr
    echo "[entrypoint] Starting vLLM serve..."
    exec vllm serve /models/vibevoice-asr \
      --served-model-name vibevoice \
      --trust-remote-code \
      --dtype bfloat16 \
      --max-num-seqs 16 \
      --max-model-len 32768 \
      --gpu-memory-utilization 0.85 \
      --num-gpu-blocks-override 4096 \
      --no-enable-prefix-caching \
      --enable-chunked-prefill \
      --chat-template-content-format openai \
      --allowed-local-media-path /app \
      --media-io-kwargs "{\"audio\": {\"target_sr\": 24000}}" \
      --enforce-eager \
      --skip-mm-profiling \
      --host 0.0.0.0 \
      --port 8000
    '

This config runs the vllm/vllm-openai:v0.14.1 image with Microsoft’s VibeVoice plugin patches applied at startup, serving weights pre-mounted at /models/vibevoice-asr so cold starts skip the 9.2 GB Hugging Face download. The server runs in eager mode with a 32k context and up to 16 concurrent sequences, exposing the model as vibevoice on the chat completions endpoint.

Flags

The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:

Flag	Value	What it does
`--trust-remote-code`	(no value)	Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures).
`--dtype`	`bfloat16`	Weight precision loaded at runtime. bfloat16: BF16 weights, no quantization.
`--max-num-seqs`	`16`	Maximum number of concurrent sequences in the batch.
`--max-model-len`	`32768`	Maximum context length (tokens) the server accepts per request.
`--gpu-memory-utilization`	`0.85`	Fraction of GPU memory vLLM may use for weights and KV cache.
`--num-gpu-blocks-override`	`4096`	Overrides vLLM’s profiled KV cache size with a fixed number of GPU blocks.
`--no-enable-prefix-caching`	(no value)	Disable prefix caching, so repeated prompts do not reuse cached KV blocks.
`--enable-chunked-prefill`	(no value)	Process long prompts in chunks so decode requests keep running.
`--chat-template-content-format`	`openai`	Format the chat template uses to render message content. openai: OpenAI-style content parts (list of typed segments) rather than a plain string.
`--allowed-local-media-path`	`/app`	Filesystem path the server may read local media files from when resolving multimodal inputs.
`--media-io-kwargs`	`{"audio": {"target_sr": 24000}}`	Options passed to the multimodal media loaders as a JSON object, for example the target sample rate for audio inputs.
`--enforce-eager`	(no value)	Run the model in eager mode instead of capturing CUDA graphs.
`--skip-mm-profiling`	(no value)	Skip multimodal memory profiling at startup, reducing startup time.

Deploy

Push the config to Baseten:

uvx truss push

You should see output similar to:

✨ Model vibevoice-asr-latency was successfully pushed ✨

   Model ID:      abc1d2ef
   Deployment ID: xyz123
   Endpoint:      model-abc1d2ef.api.baseten.co
   Logs:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

Your model ID is printed in the truss push output (abcd1234 in the example). Use it wherever you see {model_id} in the next section.

Call the model

Your deployment serves an OpenAI-compatible chat completions API at /v1/chat/completions that accepts audio inputs. Replace {model_id} with your model ID and make sure BASETEN_API_KEY is set. Send audio as an audio_url content item on a chat message. The model returns the transcription as the assistant message content.

Python
cURL

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="vibevoice",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "audio_url",
                    "audio_url": {
                        "url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav"
                    },
                }
            ],
        }
    ],
)

print(response.choices[0].message.content)

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "vibevoice",
    "messages": [
      {"role": "user", "content": [
        {"type": "audio_url", "audio_url": {"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav"}}
      ]}
    ]
  }'

​Setup

Hardware

Engine

Context

Concurrency

​Write the config

​Flags

​Deploy

​Call the model

Setup

Write the config

Flags

Deploy

Call the model