Baseten Realtime Voice is in Private Preview. Join the waitlist
to get it enabled for your organization.
x_-prefixed fields. For an overview, see Baseten Realtime Voice. For examples, see the recipes.
/websocket. The endpoint returns HTTP 400 when a client appends /realtime. All messages are JSON text frames. Audio is base64-encoded in the JSON.
Parameters
The ID of your Baseten Realtime Voice Chain. Baseten provides it when it enables Realtime Voice for your organization.
Your Baseten API key, passed as
Authorization: Bearer $BASETEN_API_KEY. Api-Key is also accepted as the scheme. This header is the only way to authenticate, so browser apps must connect through a backend that holds the key.Session flow
- Open the WebSocket and wait for
session.created. - Send
session.updatewith instructions, tools, and audio formats. - Stream audio with
input_audio_buffer.append. The agent detects the end of each turn and replies. - Play
response.output_audio.deltaaudio in order. - When the model calls a client tool, send the result with
conversation.item.create, thenresponse.create.
session.created then. Set your client’s timeout to wait for it. The server closes with code 1013 if the models don’t start in time or the replica is full, and 1011 if startup fails. Retry a 1013 close with exponential backoff.
Limits
- Idle sessions: close after 300 seconds without speech from the caller or the agent; set
x_idle_timeout_secsto change this.expires_atin thesessionobject isn’t enforced. - Client tool calls: time out after 30 seconds.
- Model Context Protocol (MCP) calls: time out after 30 seconds by default; set
x_call_timeout_secsorcall_timeout_secsper server.
Client events
session.update
Changes the session configuration. Omitted fields keep their current value. The server replies withsession.updated, carrying the effective configuration.
The server rejects the whole update when a field is invalid, returning an error event whose param names the field. It ignores unknown fields, including audio.output.speed, audio.input.noise_reduction, reasoning, parallel_tool_calls, prompt, and truncation.
realtime for a conversation, or transcription to transcribe turns without replying. An update without type keeps the current one; any other value rejects the update.A transcription session still runs voice activity detection and speech-to-text and sends the input audio and transcription events, but the agent never replies: the server sends no response.* events, and response.create gets an error with code unsupported_event. Its session object is a realtime.transcription_session, with audio.input but no instructions, voice, or tools.Function tools (
type: "function"), which run on the client, and MCP tools (type: "mcp"), which the server runs.Format of the audio you send.
- type (
string, default="audio/pcm"):audio/pcm(16-bit mono PCM),audio/pcmu(G.711 μ-law, 8 kHz), oraudio/pcma(G.711 A-law, 8 kHz). - rate (
integer, default=24000): Sample rate in Hz, foraudio/pcm. Any positive rate works, for example16000or48000: the server converts to and from its own rate. G.711 is always 8 kHz.
Voice name, or an object with
id or name. Names depend on the deployment’s TTS model and are case-sensitive; see Voice and transcription settings. x_providers.tts.voices lists the available voices, or is null when the TTS model can’t list them. A name the model doesn’t have produces replies with no audio. OpenAI voice names that the deployment doesn’t offer leave the current voice unchanged.Turn detection settings.
null is manual mode: send input_audio_buffer.commit to end a turn and response.create for the reply. Manual mode isn’t available while delegation is enabled.- type (
string, default="server_vad"):server_vadorsemantic_vad. - create_response (
boolean, default=true): Whether the agent replies when a turn ends. - interrupt_response (
boolean, default=true): Whether the caller can interrupt the agent. - threshold (
float): Withserver_vad, speech detection confidence, from 0.0 to 1.0. - silence_duration_ms (
integer): Withserver_vad, the silence before the end-of-turn check. The maximum is 3,500 ms. - eagerness (
string): Withsemantic_vad,low,medium,high, orauto.
threshold: 0.5, silence_duration_ms: 500, eagerness: auto) leaves the deployment’s tuning unchanged. session.updated reports the deployment’s live values, so silence_duration_ms can differ from 500. It echoes prefix_padding_ms as you sent it, but the value only shifts audio_start_ms; idle_timeout_ms isn’t applied.Speech-to-text settings.
null disables transcription events.- language (
string): In the STT model’s format, for exampleEnglishfor Qwen3-ASR orenfor Whisper. The server passes it to the model unchanged. Unlikesession.language, it sets what speech-to-text listens for. - model (
string): An STT route fromx_providers.stt.available. The server ignores other names. - prompt (
string) and keywords (string[]): Bias transcription toward your domain’s terms. The server applies these whenx_providers.stt.supports_contextistrue.
ISO 639-1 code used to split replies into sentences for speech. The server reads a regional code such as
es-MX as es, and ignores a code it doesn’t support. It doesn’t affect transcription: set audio.input.transcription.language for that.Whether the caller can interrupt the agent. In the same update it overrides
turn_detection.interrupt_response; across updates, the later one applies.Switches or tunes the LLM, starting with the next reply.
llm is an alias, read only when x_llm isn’t sent.- provider (
string): An LLM fromx_providers.llm.available.internalselects the Chain’s own LLM and needs noapi_key. - base_url (
string): Any other OpenAI-compatible HTTPS endpoint, such ashttps://inference.baseten.co/v1for Model APIs. This option requiresapi_keyand can’t be combined withprovider. - api_key (
string): Your key for the LLM. - model (
string): Model name sent to the provider. - reasoning_effort (
string):low,medium, orhigh. The server ignores other values, includingnone. - temperature, top_p, max_tokens, max_completion_tokens, frequency_penalty, presence_penalty, seed: Sampling parameters.
- extra (
object): Extra fields for the completion request. - server_tools (
string[]): Server-side tools for Model APIs, for example["baseten__parallel__web_search"]. - server_tool_settings (
object): Loop limits forserver_tools. See request configuration.
Tunes text-to-speech.
tts is an alias, read only when x_tts isn’t sent.- voice (
string): Voice name. In the same update it overridesaudio.output.voice; across updates, the later one applies. - split_granularity (
string):sentenceorclause.
Switches or tunes speech-to-text.
stt is an alias, read only when x_stt isn’t sent. In the same update, each field overrides the matching one in audio.input.transcription.- provider (
string): An STT route fromx_providers.stt.available. Only before any caller audio is sent. - language, prompt, keywords: As in
audio.input.transcription. - tool_keywords (
boolean, default=true): Whether terms from tool results and the agent’s replies also bias transcription.
MCP servers the platform runs, in a Baseten format. A new list replaces the previous one. Prefer MCP tools in
tools: those report discovery as an mcp_list_tools item, support approvals, and appear in the session.updated echo. Servers listed here do none of that (their tool names appear in x_providers.mcp_tools only), and type defaults to sse rather than streamable_http. Use this field when you need timeout or sse_read_timeout.- url (
string): HTTPS server URL on a public host. - type (
string, default="sse"):sseorstreamable_http. - headers (
object): HTTP headers, such asAuthorization. - tools_filter (
string[]): Only expose these tools. - server_label (
string, default="mcp"): Label on the server’smcp_callitems. - call_timeout_secs (
float, default=30): Per-call timeout. - timeout (
float, default=10) and sse_read_timeout (float, default=300): Connection and read timeouts in seconds.
Seconds without speech before the server closes the session, from
10 to 3600. Speech from the caller or the agent restarts the countdown, so each turn resets it; text events alone don’t. The timeout can’t be turned off. A new value applies immediately and restarts the countdown. session.created and session.updated report the value in effect. This is separate from turn_detection.idle_timeout_ms, which isn’t applied.To check on a quiet caller before the session closes, track the time since the last turn in your client and send a response.create with instructions such as “Ask the caller whether they’re still there.” The agent’s spoken reply restarts the countdown.Start the next response automatically after a tool result, without
response.create.Whether the agent says a short acknowledgment, such as “Let me check on that”, when a tool call takes a moment.
Whether the agent adds voice-call guidance, such as short replies and no lists, to your
instructions. Set false if your instructions set their own style.Hands over tool calls and decisions to a stronger model.
delegation is also accepted. See Delegation.MCP tools
Anmcp entry in tools connects to a remote MCP server whose tools the server calls directly:
never, always, or a filter such as {"never": {"tool_names": ["search_docs"]}}. See Approvals.mcp_list_tools item and each call as an mcp_call item.
Approvals
Withrequire_approval, a call waits for the client. The response ends with an mcp_approval_request item:
mcp_approval_response item, then send response.create:
Delegation
Delegation hands over the session’s tool calls and decisions to a stronger model on Model APIs while the voice model keeps the conversation. It uses your Baseten API key.false turns it off, and true turns it back on with its last settings. session.updated reports the settings in x_providers.delegation, without the key.
none, low, medium, or high. Any other value rejects the update. Unlike x_llm.reasoning_effort, it accepts none.function_call items; reply with function_call_output, without response.create. The server reports handoffs as mcp_call items with server_label: "delegation", and the delegate’s steps with server_label: "backend".
When the delegate’s last spoken line asks the caller a question, the caller’s next spoken turn goes straight to the delegate, without the voice model. This continues for as long as each of its replies ends with a question, so a delegate that closes every answer with “Is there anything else?” answers the rest of the call. See Delegation in the prompting guide.
The delegation field uses the Responses API format, with responses.model, responses.reasoning.effort, and responses.max_output_tokens. If you send both fields, x_delegation takes precedence.
Use your own tools and keys
Onesession.update can combine client tools, an MCP server, and an LLM endpoint, using separate keys for the server and the LLM:
{"provider": "internal"}.
input_audio_buffer.append
input_audio_buffer.commit
In manual mode, ends the caller’s turn. The server replies withinput_audio_buffer.committed and the turn’s transcript events. A commit with less than 100 ms of new audio returns an error with code input_audio_buffer_commit_empty.
input_audio_buffer.clear
Clears buffered input audio. The server replies withinput_audio_buffer.cleared.
output_audio_buffer.clear
Stops the agent’s audio, as a barge-in does.conversation.item.create
Adds an item to the conversation. The server replies withconversation.item.created, or an error with code item_not_found when no call or approval is waiting for the ID.
For
message items: input_text or input_audio parts for user, input_text for system, and output_text for assistant.For
mcp_approval_response items: the request’s id, with approve (boolean) and an optional reason.conversation.item.retrieve
The server replies withconversation.item.retrieved. Audio isn’t kept, so audio parts carry only their transcripts.
response.create
Asks the agent to respond. A request sent during a response runs after it.auto adds the response to the conversation. none runs it out of band, without tools.response.cancel
Stops the in-progress response and its audio. With no response in progress, the server replies with anerror with code response_cancel_not_active.
conversation.item.truncate
Trims an assistant item to the audio the caller heard and cancels the in-progress response. The server replies withconversation.item.truncated. Unlike OpenAI Realtime, you don’t need a separate response.cancel.
conversation.item.delete
Removes an item from the conversation. Deleting afunction_call or mcp_call item also removes its result. The server replies with conversation.item.deleted.
Server events
Session
The
session object’s x_providers block reports the effective configuration:
Provider, the current
voice, and voices: every name audio.output.voice accepts, or null when the TTS model can’t list them. Read it from session.created to see the available voices before you pick one.Delegation settings, or
null when it’s off.session object also carries x_server_version:
The voice agent release the deployment runs, such as
0.1.0, or unknown when it isn’t recorded. Read-only; include it when you report an issue.Input audio
User transcription
Conversation
Response
The server sends these events for a spoken response, in order:response.createdresponse.output_item.added, thenresponse.content_part.addedresponse.output_audio_transcript.deltaandresponse.output_audio.delta, interleavedresponse.output_audio_transcript.done, thenresponse.output_audio.doneresponse.content_part.done, thenresponse.output_item.doneresponse.done
response.output_text.delta and response.output_text.done instead of the audio and transcript events.
response.done carries a status of completed, cancelled, or incomplete. For a cancelled response, status_details.reason is turn_detected when the caller interrupted, or client_cancelled. A response that hit its token cap is incomplete with reason max_output_tokens.
Function calls
Tools the agent runs
The server reports MCP tools, server-side tools, and delegation asmcp_call items, which need no reply:
An MCP call can outlast the response that made it. That response ends (
response.done) with the item still in progress, and the call finishes later under the same item ID: response.mcp_call.completed or response.mcp_call.failed, then conversation.item.done with the item’s output or error. Read the result from whichever arrives, not from response.done. The agent then answers with the result in a new response.
server_label names the MCP server, the server-side tool’s provider, delegation, or backend.
Errors
error reports a problem with an event you sent, or a failed model call during a turn. It echoes the client’s event_id, and param names the field at fault when there is one. A session.update that fails applies none of its changes.
The server reports a failed transcription as
conversation.item.input_audio_transcription.failed, not as error.
The server closes the WebSocket with 1013 when the models aren’t ready in time or the deployment is at capacity: reconnect with backoff. It closes with 1011 when the models fail to start or a session fails on the server.
Latency profile
response.done and the first response.output_audio.delta of each turn carry x_profile timings in seconds:
LLM start to first token.
null when the turn used a speculative reply: the agent starts writing a reply when the caller pauses and sends it if the turn ends there, so the reply has no separate first-token time.x_profile fields are for diagnostics only and can change between releases.