Skip to main content
Baseten Realtime Voice is in Private Preview. Join the waitlist to get it enabled for your organization.
Baseten Realtime Voice runs speech-to-text (STT), a large language model (LLM), and text-to-speech (TTS) behind a single WebSocket that implements the OpenAI Realtime event protocol. Baseten-specific options use x_-prefixed fields. For an overview, see Baseten Realtime Voice. For examples, see the recipes.
The path must end in /websocket. The endpoint returns HTTP 400 when a client appends /realtime. All messages are JSON text frames. Audio is base64-encoded in the JSON.

Parameters

string
required
The ID of your Baseten Realtime Voice Chain. Baseten provides it when it enables Realtime Voice for your organization.
string
required
The environment to connect to, for example production.
string
The deployment to connect to, in place of env_name.
string
required
Your Baseten API key, passed as Authorization: Bearer $BASETEN_API_KEY. Api-Key is also accepted as the scheme. This header is the only way to authenticate, so browser apps must connect through a backend that holds the key.

Session flow

  1. Open the WebSocket and wait for session.created.
  2. Send session.update with instructions, tools, and audio formats.
  3. Stream audio with input_audio_buffer.append. The agent detects the end of each turn and replies.
  4. Play response.output_audio.delta audio in order.
  5. When the model calls a client tool, send the result with conversation.item.create, then response.create.
If the models are still starting, the server holds the connection until they’re ready, for up to 45 minutes, and sends session.created then. Set your client’s timeout to wait for it. The server closes with code 1013 if the models don’t start in time or the replica is full, and 1011 if startup fails. Retry a 1013 close with exponential backoff.

Limits

  • Idle sessions: close after 300 seconds without speech from the caller or the agent; set x_idle_timeout_secs to change this. expires_at in the session object isn’t enforced.
  • Client tool calls: time out after 30 seconds.
  • Model Context Protocol (MCP) calls: time out after 30 seconds by default; set x_call_timeout_secs or call_timeout_secs per server.

Client events

session.update

Changes the session configuration. Omitted fields keep their current value. The server replies with session.updated, carrying the effective configuration. The server rejects the whole update when a field is invalid, returning an error event whose param names the field. It ignores unknown fields, including audio.output.speed, audio.input.noise_reduction, reasoning, parallel_tool_calls, prompt, and truncation.
string
default:"realtime"
realtime for a conversation, or transcription to transcribe turns without replying. An update without type keeps the current one; any other value rejects the update.A transcription session still runs voice activity detection and speech-to-text and sends the input audio and transcription events, but the agent never replies: the server sends no response.* events, and response.create gets an error with code unsupported_event. Its session object is a realtime.transcription_session, with audio.input but no instructions, voice, or tools.
string
The system prompt that defines the agent’s behavior.
array
Function tools (type: "function"), which run on the client, and MCP tools (type: "mcp"), which the server runs.
string | object
default:"auto"
auto, none, required, or {"type": "function", "name": "<tool name>"}.
integer | string
default:"inf"
Token cap for each reply, or "inf".
object
Format of the audio you send.
  • type (string, default="audio/pcm"): audio/pcm (16-bit mono PCM), audio/pcmu (G.711 μ-law, 8 kHz), or audio/pcma (G.711 A-law, 8 kHz).
  • rate (integer, default=24000): Sample rate in Hz, for audio/pcm. Any positive rate works, for example 16000 or 48000: the server converts to and from its own rate. G.711 is always 8 kHz.
object
Format of the audio the server sends. Same fields as audio.input.format.
string | object
Voice name, or an object with id or name. Names depend on the deployment’s TTS model and are case-sensitive; see Voice and transcription settings. x_providers.tts.voices lists the available voices, or is null when the TTS model can’t list them. A name the model doesn’t have produces replies with no audio. OpenAI voice names that the deployment doesn’t offer leave the current voice unchanged.
object | null
Turn detection settings. null is manual mode: send input_audio_buffer.commit to end a turn and response.create for the reply. Manual mode isn’t available while delegation is enabled.
  • type (string, default="server_vad"): server_vad or semantic_vad.
  • create_response (boolean, default=true): Whether the agent replies when a turn ends.
  • interrupt_response (boolean, default=true): Whether the caller can interrupt the agent.
  • threshold (float): With server_vad, speech detection confidence, from 0.0 to 1.0.
  • silence_duration_ms (integer): With server_vad, the silence before the end-of-turn check. The maximum is 3,500 ms.
  • eagerness (string): With semantic_vad, low, medium, high, or auto.
Setting these OpenAI Realtime defaults (threshold: 0.5, silence_duration_ms: 500, eagerness: auto) leaves the deployment’s tuning unchanged. session.updated reports the deployment’s live values, so silence_duration_ms can differ from 500. It echoes prefix_padding_ms as you sent it, but the value only shifts audio_start_ms; idle_timeout_ms isn’t applied.
object | null
Speech-to-text settings. null disables transcription events.
  • language (string): In the STT model’s format, for example English for Qwen3-ASR or en for Whisper. The server passes it to the model unchanged. Unlike session.language, it sets what speech-to-text listens for.
  • model (string): An STT route from x_providers.stt.available. The server ignores other names.
  • prompt (string) and keywords (string[]): Bias transcription toward your domain’s terms. The server applies these when x_providers.stt.supports_context is true.
string[]
default:"[\"audio\"]"
["audio"] for spoken replies, or ["text"] for text-only replies.
string
ISO 639-1 code used to split replies into sentences for speech. The server reads a regional code such as es-MX as es, and ignores a code it doesn’t support. It doesn’t affect transcription: set audio.input.transcription.language for that.
boolean
default:"true"
Whether the caller can interrupt the agent. In the same update it overrides turn_detection.interrupt_response; across updates, the later one applies.
object
Switches or tunes the LLM, starting with the next reply. llm is an alias, read only when x_llm isn’t sent.
  • provider (string): An LLM from x_providers.llm.available. internal selects the Chain’s own LLM and needs no api_key.
  • base_url (string): Any other OpenAI-compatible HTTPS endpoint, such as https://inference.baseten.co/v1 for Model APIs. This option requires api_key and can’t be combined with provider.
  • api_key (string): Your key for the LLM.
  • model (string): Model name sent to the provider.
  • reasoning_effort (string): low, medium, or high. The server ignores other values, including none.
  • temperature, top_p, max_tokens, max_completion_tokens, frequency_penalty, presence_penalty, seed: Sampling parameters.
  • extra (object): Extra fields for the completion request.
  • server_tools (string[]): Server-side tools for Model APIs, for example ["baseten__parallel__web_search"].
  • server_tool_settings (object): Loop limits for server_tools. See request configuration.
object
Tunes text-to-speech. tts is an alias, read only when x_tts isn’t sent.
  • voice (string): Voice name. In the same update it overrides audio.output.voice; across updates, the later one applies.
  • split_granularity (string): sentence or clause.
object
Switches or tunes speech-to-text. stt is an alias, read only when x_stt isn’t sent. In the same update, each field overrides the matching one in audio.input.transcription.
  • provider (string): An STT route from x_providers.stt.available. Only before any caller audio is sent.
  • language, prompt, keywords: As in audio.input.transcription.
  • tool_keywords (boolean, default=true): Whether terms from tool results and the agent’s replies also bias transcription.
array
MCP servers the platform runs, in a Baseten format. A new list replaces the previous one. Prefer MCP tools in tools: those report discovery as an mcp_list_tools item, support approvals, and appear in the session.updated echo. Servers listed here do none of that (their tool names appear in x_providers.mcp_tools only), and type defaults to sse rather than streamable_http. Use this field when you need timeout or sse_read_timeout.
  • url (string): HTTPS server URL on a public host.
  • type (string, default="sse"): sse or streamable_http.
  • headers (object): HTTP headers, such as Authorization.
  • tools_filter (string[]): Only expose these tools.
  • server_label (string, default="mcp"): Label on the server’s mcp_call items.
  • call_timeout_secs (float, default=30): Per-call timeout.
  • timeout (float, default=10) and sse_read_timeout (float, default=300): Connection and read timeouts in seconds.
number
default:"300"
Seconds without speech before the server closes the session, from 10 to 3600. Speech from the caller or the agent restarts the countdown, so each turn resets it; text events alone don’t. The timeout can’t be turned off. A new value applies immediately and restarts the countdown. session.created and session.updated report the value in effect. This is separate from turn_detection.idle_timeout_ms, which isn’t applied.To check on a quiet caller before the session closes, track the time since the last turn in your client and send a response.create with instructions such as “Ask the caller whether they’re still there.” The agent’s spoken reply restarts the countdown.
boolean
default:"false"
Start the next response automatically after a tool result, without response.create.
boolean
default:"true"
Whether the agent says a short acknowledgment, such as “Let me check on that”, when a tool call takes a moment.
boolean
default:"true"
Whether the agent adds voice-call guidance, such as short replies and no lists, to your instructions. Set false if your instructions set their own style.
boolean | object
Hands over tool calls and decisions to a stronger model. delegation is also accepted. See Delegation.

MCP tools

An mcp entry in tools connects to a remote MCP server whose tools the server calls directly:
string
required
Label on the server’s mcp_list_tools and mcp_call items.
string
required
HTTPS server URL on a public host.
string
Sent to the server as a Bearer token.
object
Other HTTP headers sent to the server.
string[]
Only expose these tools.
string | object
default:"never"
never, always, or a filter such as {"never": {"tool_names": ["search_docs"]}}. See Approvals.
string
default:"streamable_http"
sse for an SSE server.
float
default:"30"
Per-call timeout in seconds.
The server reports discovery as an mcp_list_tools item and each call as an mcp_call item.

Approvals

With require_approval, a call waits for the client. The response ends with an mcp_approval_request item:
Answer with an mcp_approval_response item, then send response.create:
The server doesn’t run a rejected request, or one left unanswered for 5 minutes.

Delegation

Delegation hands over the session’s tool calls and decisions to a stronger model on Model APIs while the voice model keeps the conversation. It uses your Baseten API key.
false turns it off, and true turns it back on with its last settings. session.updated reports the settings in x_providers.delegation, without the key.
string
Your Baseten API key. Required to enable delegation.
string
Model APIs model. Required the first time delegation is turned on.
integer
Token cap for each delegate reply.
string
default:"none"
none, low, medium, or high. Any other value rejects the update. Unlike x_llm.reasoning_effort, it accepts none.
While delegation is enabled, the delegate’s tool calls arrive as function_call items; reply with function_call_output, without response.create. The server reports handoffs as mcp_call items with server_label: "delegation", and the delegate’s steps with server_label: "backend". When the delegate’s last spoken line asks the caller a question, the caller’s next spoken turn goes straight to the delegate, without the voice model. This continues for as long as each of its replies ends with a question, so a delegate that closes every answer with “Is there anything else?” answers the rest of the call. See Delegation in the prompting guide. The delegation field uses the Responses API format, with responses.model, responses.reasoning.effort, and responses.max_output_tokens. If you send both fields, x_delegation takes precedence.

Use your own tools and keys

One session.update can combine client tools, an MCP server, and an LLM endpoint, using separate keys for the server and the LLM:
Keys sent this way apply to the session only. To switch back to the Chain’s own LLM, send {"provider": "internal"}.

input_audio_buffer.append

string
required
Base64-encoded audio in the format set by audio.input.format.

input_audio_buffer.commit

In manual mode, ends the caller’s turn. The server replies with input_audio_buffer.committed and the turn’s transcript events. A commit with less than 100 ms of new audio returns an error with code input_audio_buffer_commit_empty.

input_audio_buffer.clear

Clears buffered input audio. The server replies with input_audio_buffer.cleared.

output_audio_buffer.clear

Stops the agent’s audio, as a barge-in does.

conversation.item.create

Adds an item to the conversation. The server replies with conversation.item.created, or an error with code item_not_found when no call or approval is waiting for the ID.
string
required
message, function_call_output, or mcp_approval_response.
string
default:"user"
For message items: user, assistant, or system.
array
For message items: input_text or input_audio parts for user, input_text for system, and output_text for assistant.
string
For function_call_output items: the call’s call_id.
string | object
For function_call_output items: the tool result.
string
For mcp_approval_response items: the request’s id, with approve (boolean) and an optional reason.

conversation.item.retrieve

string
required
ID of the item to retrieve.
The server replies with conversation.item.retrieved. Audio isn’t kept, so audio parts carry only their transcripts.

response.create

Asks the agent to respond. A request sent during a response runs after it.
string
Extra instructions for this response.
array
message items, or {"type": "item_reference", "id": "<item id>"}, to add before responding.
string | object
Tool choice for this response.
integer | string
Token cap for this response.
string[]
["text"] or ["audio"].
string
default:"auto"
auto adds the response to the conversation. none runs it out of band, without tools.
object
The server echoes this on the response.

response.cancel

Stops the in-progress response and its audio. With no response in progress, the server replies with an error with code response_cancel_not_active.

conversation.item.truncate

Trims an assistant item to the audio the caller heard and cancels the in-progress response. The server replies with conversation.item.truncated. Unlike OpenAI Realtime, you don’t need a separate response.cancel.
string
required
ID of the assistant item being played.
integer
default:"0"
Always 0.
integer
Milliseconds of the item’s audio the caller heard.

conversation.item.delete

Removes an item from the conversation. Deleting a function_call or mcp_call item also removes its result. The server replies with conversation.item.deleted.
string
required
ID of the item to delete.

Server events

Session

The session object’s x_providers block reports the effective configuration:
object
Provider, model, server_tools, the default provider, and the available providers.
object
Provider, the current voice, and voices: every name audio.output.voice accepts, or null when the TTS model can’t list them. Read it from session.created to see the available voices before you pick one.
object
Route, available routes, supports_context, tool_keywords, and learned_keywords.
string[]
Registered client tool names.
string[]
Registered MCP tool names.
object | null
Delegation settings, or null when it’s off.
The session object also carries x_server_version:
string
The voice agent release the deployment runs, such as 0.1.0, or unknown when it isn’t recorded. Read-only; include it when you report an issue.

Input audio

User transcription

Conversation

Response

The server sends these events for a spoken response, in order:
  1. response.created
  2. response.output_item.added, then response.content_part.added
  3. response.output_audio_transcript.delta and response.output_audio.delta, interleaved
  4. response.output_audio_transcript.done, then response.output_audio.done
  5. response.content_part.done, then response.output_item.done
  6. response.done
For a text-only response, the server sends response.output_text.delta and response.output_text.done instead of the audio and transcript events. response.done carries a status of completed, cancelled, or incomplete. For a cancelled response, status_details.reason is turn_detected when the caller interrupted, or client_cancelled. A response that hit its token cap is incomplete with reason max_output_tokens.

Function calls

Tools the agent runs

The server reports MCP tools, server-side tools, and delegation as mcp_call items, which need no reply: An MCP call can outlast the response that made it. That response ends (response.done) with the item still in progress, and the call finishes later under the same item ID: response.mcp_call.completed or response.mcp_call.failed, then conversation.item.done with the item’s output or error. Read the result from whichever arrives, not from response.done. The agent then answers with the result in a new response. server_label names the MCP server, the server-side tool’s provider, delegation, or backend.

Errors

error reports a problem with an event you sent, or a failed model call during a turn. It echoes the client’s event_id, and param names the field at fault when there is one. A session.update that fails applies none of its changes. The server reports a failed transcription as conversation.item.input_audio_transcription.failed, not as error. The server closes the WebSocket with 1013 when the models aren’t ready in time or the deployment is at capacity: reconnect with backoff. It closes with 1011 when the models fail to start or a session fails on the server.

Latency profile

response.done and the first response.output_audio.delta of each turn carry x_profile timings in seconds:
float
End of speech to final transcript.
float
Final transcript to LLM start.
float | null
LLM start to first token. null when the turn used a speculative reply: the agent starts writing a reply when the caller pauses and sends it if the turn ends there, so the reply has no separate first-token time.
float
First token to first audio.
float
End of speech to first audio.
Other x_profile fields are for diagnostics only and can change between releases.