Skip to main content
Baseten Realtime Voice is in Private Preview. Join the waitlist to get it enabled for your organization.
Each recipe is a session.update body to send after the WebSocket connects, plus the events it requires. Every field is optional and an update changes only the fields it sends, so combine recipes by merging their session objects. Each recipe sends "type": "realtime". The server doesn’t need it, but the OpenAI Agents SDK checks raw events against OpenAI’s schema, which does. Some recipes connect to a remote Model Context Protocol (MCP) server. Another switches the large language model (LLM) for each turn. For every field and its limits, see the Baseten Realtime Voice API reference. Replace <BASETEN_API_KEY> with your Baseten API key. Keys sent in a session apply to that session only and are never echoed in session.updated.

Phone support agent

This recipe sets a persona with phone-style instructions, server voice activity detection (VAD) that lets the caller talk over the agent, and key terms that help speech-to-text (STT) hear product names:
Baseten Realtime Voice adds voice-call guidance after your instructions, such as short replies and one question at a time, so your instructions don’t need to cover speaking style. To have the agent speak first, ask for a greeting. Per-response instructions are added to the session’s, so it stays in character:

Telephony audio

Send and receive G.711 μ-law at 8 kHz, the format that phone media streams use, with no transcoding on your side:
Use audio/pcma for A-law. With audio/pcmu, each input_audio_buffer.append carries base64 μ-law audio, and response.output_audio.delta returns it. With audio/pcma, both carry A-law audio.

Voice and transcription settings

Pick a voice and set the transcription language and key terms:
Voice names depend on the deployment’s TTS model. A Qwen3-TTS deployment speaks named voices, such as ryan. A Magpie deployment speaks names such as Magpie-Multilingual.EN-US.Mia, some with an emotion (Magpie-Multilingual.EN-US.Mia.Calm); which emotions a speaker has varies. Names are case-sensitive, and a name the model doesn’t have produces replies with no audio. To see the voices a deployment offers, read session.x_providers.tts in the session.created event, which arrives as soon as the WebSocket connects; every session.updated repeats it. voice is the current voice, and voices lists every name audio.output.voice accepts:
voices is null when the TTS model can’t list its voices. x_providers.stt.available lists the STT routes the same way. To switch the STT route, send "x_stt": {"provider": "<route>"} before any caller audio. The language format depends on the STT model: English for Qwen3-ASR, en for Whisper.

Push-to-talk

Turn off automatic turn detection so the client decides when the caller is done:
Stream audio with input_audio_buffer.append while the button is held. On release, end the turn and ask for the reply:

Client tools

Function tools run in your application, next to your data. The agent speaks a short acknowledgment if a tool takes more than a moment:
When the model calls a tool, the server sends response.function_call_arguments.done with name, call_id, and arguments. Run the tool, return the result, and then ask for the next response:
Set x_auto_continue: true to have the server start the next response when the result arrives.

MCP server

Baseten Realtime Voice connects to a remote MCP server and runs its tools directly, without a client round trip. This example uses DeepWiki’s public server, which answers questions about GitHub repositories:
For a server that needs a key, add "authorization": "<MCP_API_KEY>". The server URL must use HTTPS on a public host. An mcp_list_tools item and x_providers.mcp_tools report the server’s tools. Check them after the update: a name in allowed_tools that the server doesn’t offer registers nothing, and the agent then can’t make the call it promises. A call can outlast the response that made it. That response ends with the mcp_call still in progress, and the call finishes later under the same item ID: read its output from conversation.item.done, not from response.done. See Tools the agent runs. To approve each call first, set "require_approval": "always". The response then ends with an mcp_approval_request item. Answer it, then ask for the next response:

Delegation

A fast voice model keeps the conversation while a stronger model on Model APIs does the work: it reads the policy in your instructions, calls your tools, and writes what the caller hears. The session names the delegate’s model and sends your Baseten API key. Baseten bills the usage to your account:
The delegate’s lookup_account calls arrive as function_call items; reply with function_call_output, and the delegate continues on its own. The server reports each handoff as an mcp_call item with server_label: "delegation". To turn delegation off:

Switch the LLM per turn

Route everyday turns to the Chain’s own LLM and harder ones to a larger model on Model APIs. Turn off create_response so the server detects and transcribes each turn but waits for the client to start the reply:
When conversation.item.input_audio_transcription.completed arrives, decide which model the turn needs. For a harder turn, switch to Model APIs with your own key, then ask for the reply:
The server applies events in order, so the reply runs on the new model, and the conversation carries over. To switch back to the Chain’s own LLM, which needs no key:
session.updated confirms each switch in x_providers.llm.