Baseten Realtime Voice is in Private Preview. Join the waitlist
to get it enabled for your organization.
session.update body to send after the WebSocket connects, plus the events it requires. Every field is optional and an update changes only the fields it sends, so combine recipes by merging their session objects. Each recipe sends "type": "realtime". The server doesn’t need it, but the OpenAI Agents SDK checks raw events against OpenAI’s schema, which does. Some recipes connect to a remote Model Context Protocol (MCP) server. Another switches the large language model (LLM) for each turn. For every field and its limits, see the Baseten Realtime Voice API reference.
Replace <BASETEN_API_KEY> with your Baseten API key. Keys sent in a session apply to that session only and are never echoed in session.updated.
Phone support agent
This recipe sets a persona with phone-style instructions, server voice activity detection (VAD) that lets the caller talk over the agent, and key terms that help speech-to-text (STT) hear product names:Telephony audio
Send and receive G.711 μ-law at 8 kHz, the format that phone media streams use, with no transcoding on your side:audio/pcma for A-law. With audio/pcmu, each input_audio_buffer.append carries base64 μ-law audio, and response.output_audio.delta returns it. With audio/pcma, both carry A-law audio.
Voice and transcription settings
Pick a voice and set the transcription language and key terms:ryan. A Magpie deployment speaks names such as Magpie-Multilingual.EN-US.Mia, some with an emotion (Magpie-Multilingual.EN-US.Mia.Calm); which emotions a speaker has varies. Names are case-sensitive, and a name the model doesn’t have produces replies with no audio.
To see the voices a deployment offers, read session.x_providers.tts in the session.created event, which arrives as soon as the WebSocket connects; every session.updated repeats it. voice is the current voice, and voices lists every name audio.output.voice accepts:
voices is null when the TTS model can’t list its voices. x_providers.stt.available lists the STT routes the same way. To switch the STT route, send "x_stt": {"provider": "<route>"} before any caller audio. The language format depends on the STT model: English for Qwen3-ASR, en for Whisper.
Push-to-talk
Turn off automatic turn detection so the client decides when the caller is done:input_audio_buffer.append while the button is held. On release, end the turn and ask for the reply:
Client tools
Function tools run in your application, next to your data. The agent speaks a short acknowledgment if a tool takes more than a moment:response.function_call_arguments.done with name, call_id, and arguments. Run the tool, return the result, and then ask for the next response:
x_auto_continue: true to have the server start the next response when the result arrives.
MCP server
Baseten Realtime Voice connects to a remote MCP server and runs its tools directly, without a client round trip. This example uses DeepWiki’s public server, which answers questions about GitHub repositories:"authorization": "<MCP_API_KEY>". The server URL must use HTTPS on a public host.
An mcp_list_tools item and x_providers.mcp_tools report the server’s tools. Check them after the update: a name in allowed_tools that the server doesn’t offer registers nothing, and the agent then can’t make the call it promises.
A call can outlast the response that made it. That response ends with the mcp_call still in progress, and the call finishes later under the same item ID: read its output from conversation.item.done, not from response.done. See Tools the agent runs.
To approve each call first, set "require_approval": "always". The response then ends with an mcp_approval_request item. Answer it, then ask for the next response:
Delegation
A fast voice model keeps the conversation while a stronger model on Model APIs does the work: it reads the policy in yourinstructions, calls your tools, and writes what the caller hears. The session names the delegate’s model and sends your Baseten API key. Baseten bills the usage to your account:
lookup_account calls arrive as function_call items; reply with function_call_output, and the delegate continues on its own. The server reports each handoff as an mcp_call item with server_label: "delegation". To turn delegation off:
Switch the LLM per turn
Route everyday turns to the Chain’s own LLM and harder ones to a larger model on Model APIs. Turn offcreate_response so the server detects and transcribes each turn but waits for the client to start the reply:
conversation.item.input_audio_transcription.completed arrives, decide which model the turn needs. For a harder turn, switch to Model APIs with your own key, then ask for the reply:
session.updated confirms each switch in x_providers.llm.