> ## Documentation Index
> Fetch the complete documentation index at: https://docs.baseten.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Baseten Realtime Voice API reference

> Run a speech-to-speech voice agent with Baseten Realtime Voice over a WebSocket using OpenAI Realtime-compatible events.

> [!NOTE]
>
> Baseten Realtime Voice is in **Private Preview**. [Join the waitlist](https://baseten.co/talk-to-us/voice-beta-launch)
> to get it enabled for your organization.

Baseten Realtime Voice runs speech-to-text (STT), a large language model (LLM), and text-to-speech (TTS) behind a single WebSocket that implements the [OpenAI Realtime](https://platform.openai.com/docs/api-reference/realtime) event protocol. Baseten-specific options use `x_`-prefixed fields. For an overview, see [Baseten Realtime Voice](https://docs.baseten.co/inference/audio/realtime-voice/overview.md). For examples, see the [recipes](https://docs.baseten.co/inference/audio/realtime-voice/recipes.md).

```sh theme={"system"}
wss://chain-{chain_id}.api.baseten.co/environments/{env_name}/websocket
wss://chain-{chain_id}.api.baseten.co/deployment/{deployment_id}/websocket
wss://chain-{chain_id}.api.baseten.co/development/websocket
```

The path must end in `/websocket`. The endpoint returns HTTP 400 when a client appends `/realtime`. All messages are JSON text frames. Audio is base64-encoded in the JSON.

## Parameters

**chain\_id** `string` *required*

The ID of your Baseten Realtime Voice Chain. Baseten provides it when it enables Realtime Voice for your organization.

**env\_name** `string` *required*

The environment to connect to, for example `production`.

**deployment\_id** `string`

The deployment to connect to, in place of `env_name`.

**Authorization** `string` *required*

Your Baseten API key, passed as `Authorization: Bearer $BASETEN_API_KEY`. `Api-Key` is also accepted as the scheme. This header is the only way to authenticate, so browser apps must connect through a backend that holds the key.

## Session flow

1. Open the WebSocket and wait for `session.created`.
2. Send `session.update` with instructions, tools, and audio formats.
3. Stream audio with `input_audio_buffer.append`. The agent detects the end of each turn and replies.
4. Play `response.output_audio.delta` audio in order.
5. When the model calls a client tool, send the result with `conversation.item.create`, then `response.create`.

If the models are still starting, the server holds the connection until they're ready, for up to 45 minutes, and sends `session.created` then. Set your client's timeout to wait for it. The server closes with code `1013` if the models don't start in time or the replica is full, and `1011` if startup fails. Retry a `1013` close with exponential backoff.

## Limits

* **Idle sessions:** close after 300 seconds without speech from the caller or the agent; set [`x_idle_timeout_secs`](#param-session-x-idle-timeout-secs) to change this. `expires_at` in the `session` object isn't enforced.
* **Client tool calls:** time out after 30 seconds.
* **Model Context Protocol (MCP) calls:** time out after 30 seconds by default; set `x_call_timeout_secs` or `call_timeout_secs` per server.

## Client events

### session.update

Changes the session configuration. Omitted fields keep their current value. The server replies with `session.updated`, carrying the effective configuration.

The server rejects the whole update when a field is invalid, returning an `error` event whose `param` names the field. It ignores unknown fields, including `audio.output.speed`, `audio.input.noise_reduction`, `reasoning`, `parallel_tool_calls`, `prompt`, and `truncation`.

**session.type** `string` (default: realtime)

`realtime` for a conversation, or `transcription` to transcribe turns without replying. An update without `type` keeps the current one; any other value rejects the update.

A transcription session still runs voice activity detection and speech-to-text and sends the input audio and transcription events, but the agent never replies: the server sends no `response.*` events, and `response.create` gets an `error` with code `unsupported_event`. Its `session` object is a `realtime.transcription_session`, with `audio.input` but no instructions, voice, or tools.

**session.instructions** `string`

The system prompt that defines the agent's behavior.

**session.tools** `array`

Function tools (`type: "function"`), which run on the client, and [MCP tools](#mcp-tools) (`type: "mcp"`), which the server runs.

**session.tool\_choice** `string | object` (default: auto)

`auto`, `none`, `required`, or `{"type": "function", "name": "<tool name>"}`.

**session.max\_output\_tokens** `integer | string` (default: inf)

Token cap for each reply, or `"inf"`.

**session.audio.input.format** `object`

Format of the audio you send.

* **type** (`string`, default=`"audio/pcm"`): `audio/pcm` (16-bit mono PCM), `audio/pcmu` (G.711 μ-law, 8 kHz), or `audio/pcma` (G.711 A-law, 8 kHz).
* **rate** (`integer`, default=`24000`): Sample rate in Hz, for `audio/pcm`. Any positive rate works, for example `16000` or `48000`: the server converts to and from its own rate. G.711 is always 8 kHz.

**session.audio.output.format** `object`

Format of the audio the server sends. Same fields as `audio.input.format`.

**session.audio.output.voice** `string | object`

Voice name, or an object with `id` or `name`. Names depend on the deployment's TTS model and are case-sensitive; see [Voice and transcription settings](https://docs.baseten.co/inference/audio/realtime-voice/recipes.md#voice-and-transcription-settings). `x_providers.tts.voices` lists the available voices, or is `null` when the TTS model can't list them. A name the model doesn't have produces replies with no audio. OpenAI voice names that the deployment doesn't offer leave the current voice unchanged.

**session.audio.input.turn\_detection** `object | null`

Turn detection settings. `null` is manual mode: send `input_audio_buffer.commit` to end a turn and `response.create` for the reply. Manual mode isn't available while [delegation](#delegation) is enabled.

* **type** (`string`, default=`"server_vad"`): `server_vad` or `semantic_vad`.
* **create\_response** (`boolean`, default=`true`): Whether the agent replies when a turn ends.
* **interrupt\_response** (`boolean`, default=`true`): Whether the caller can interrupt the agent.
* **threshold** (`float`): With `server_vad`, speech detection confidence, from 0.0 to 1.0.
* **silence\_duration\_ms** (`integer`): With `server_vad`, the silence before the end-of-turn check. The maximum is 3,500 ms.
* **eagerness** (`string`): With `semantic_vad`, `low`, `medium`, `high`, or `auto`.

Setting these OpenAI Realtime defaults (`threshold: 0.5`, `silence_duration_ms: 500`, `eagerness: auto`) leaves the deployment's tuning unchanged. `session.updated` reports the deployment's live values, so `silence_duration_ms` can differ from 500. It echoes `prefix_padding_ms` as you sent it, but the value only shifts `audio_start_ms`; `idle_timeout_ms` isn't applied.

**session.audio.input.transcription** `object | null`

Speech-to-text settings. `null` disables transcription events.

* **language** (`string`): In the STT model's format, for example `English` for Qwen3-ASR or `en` for Whisper. The server passes it to the model unchanged. Unlike `session.language`, it sets what speech-to-text listens for.
* **model** (`string`): An STT route from `x_providers.stt.available`. The server ignores other names.
* **prompt** (`string`) and **keywords** (`string[]`): Bias transcription toward your domain's terms. The server applies these when `x_providers.stt.supports_context` is `true`.

**session.output\_modalities** `string[]` (default: \["audio"])

`["audio"]` for spoken replies, or `["text"]` for text-only replies.

**session.language** `string`

ISO 639-1 code used to split replies into sentences for speech. The server reads a regional code such as `es-MX` as `es`, and ignores a code it doesn't support. It doesn't affect transcription: set `audio.input.transcription.language` for that.

**session.allow\_interruptions** `boolean` (default: true)

Whether the caller can interrupt the agent. In the same update it overrides `turn_detection.interrupt_response`; across updates, the later one applies.

**session.x\_llm** `object`

Switches or tunes the LLM, starting with the next reply. `llm` is an alias, read only when `x_llm` isn't sent.

* **provider** (`string`): An LLM from `x_providers.llm.available`. `internal` selects the Chain's own LLM and needs no `api_key`.
* **base\_url** (`string`): Any other OpenAI-compatible HTTPS endpoint, such as `https://inference.baseten.co/v1` for [Model APIs](https://docs.baseten.co/inference/model-apis/overview.md). This option requires `api_key` and can't be combined with `provider`.
* **api\_key** (`string`): Your key for the LLM.
* **model** (`string`): Model name sent to the provider.
* **reasoning\_effort** (`string`): `low`, `medium`, or `high`. The server ignores other values, including `none`.
* **temperature**, **top\_p**, **max\_tokens**, **max\_completion\_tokens**, **frequency\_penalty**, **presence\_penalty**, **seed**: Sampling parameters.
* **extra** (`object`): Extra fields for the completion request.
* **server\_tools** (`string[]`): [Server-side tools](https://docs.baseten.co/reference/inference-api/server-side-tool-execution.md) for Model APIs, for example `["baseten__parallel__web_search"]`.
* **server\_tool\_settings** (`object`): Loop limits for `server_tools`. See [request configuration](https://docs.baseten.co/reference/inference-api/server-side-tool-execution.md#request-configuration).

**session.x\_tts** `object`

Tunes text-to-speech. `tts` is an alias, read only when `x_tts` isn't sent.

* **voice** (`string`): Voice name. In the same update it overrides `audio.output.voice`; across updates, the later one applies.
* **split\_granularity** (`string`): `sentence` or `clause`.

**session.x\_stt** `object`

Switches or tunes speech-to-text. `stt` is an alias, read only when `x_stt` isn't sent. In the same update, each field overrides the matching one in `audio.input.transcription`.

* **provider** (`string`): An STT route from `x_providers.stt.available`. Only before any caller audio is sent.
* **language**, **prompt**, **keywords**: As in `audio.input.transcription`.
* **tool\_keywords** (`boolean`, default=`true`): Whether terms from tool results and the agent's replies also bias transcription.

**session.mcp\_servers** `array`

MCP servers the platform runs, in a Baseten format. A new list replaces the previous one. Prefer [MCP tools](#mcp-tools) in `tools`: those report discovery as an `mcp_list_tools` item, support approvals, and appear in the `session.updated` echo. Servers listed here do none of that (their tool names appear in `x_providers.mcp_tools` only), and `type` defaults to `sse` rather than `streamable_http`. Use this field when you need `timeout` or `sse_read_timeout`.

* **url** (`string`): HTTPS server URL on a public host.
* **type** (`string`, default=`"sse"`): `sse` or `streamable_http`.
* **headers** (`object`): HTTP headers, such as `Authorization`.
* **tools\_filter** (`string[]`): Only expose these tools.
* **server\_label** (`string`, default=`"mcp"`): Label on the server's `mcp_call` items.
* **call\_timeout\_secs** (`float`, default=`30`): Per-call timeout.
* **timeout** (`float`, default=`10`) and **sse\_read\_timeout** (`float`, default=`300`): Connection and read timeouts in seconds.

**session.x\_idle\_timeout\_secs** `number` (default: 300)

Seconds without speech before the server closes the session, from `10` to `3600`. Speech from the caller or the agent restarts the countdown, so each turn resets it; text events alone don't. The timeout can't be turned off. A new value applies immediately and restarts the countdown. `session.created` and `session.updated` report the value in effect. This is separate from `turn_detection.idle_timeout_ms`, which isn't applied.

To check on a quiet caller before the session closes, track the time since the last turn in your client and send a `response.create` with `instructions` such as "Ask the caller whether they're still there." The agent's spoken reply restarts the countdown.

**session.x\_auto\_continue** `boolean` (default: false)

Start the next response automatically after a tool result, without `response.create`.

**session.tool\_acknowledgement** `boolean` (default: true)

Whether the agent says a short acknowledgment, such as "Let me check on that", when a tool call takes a moment.

**session.spoken\_style** `boolean` (default: true)

Whether the agent adds voice-call guidance, such as short replies and no lists, to your `instructions`. Set `false` if your instructions set their own style.

**session.x\_delegation** `boolean | object`

Hands over tool calls and decisions to a stronger model. `delegation` is also accepted. See [Delegation](#delegation).

#### MCP tools

An `mcp` entry in `tools` connects to a remote MCP server whose tools the server calls directly:

```json theme={"system"}
{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "tools": [{
      "type": "mcp",
      "server_label": "docs",
      "server_url": "https://mcp.example.com/mcp",
      "authorization": "<MCP_API_KEY>",
      "allowed_tools": ["search_docs"],
      "require_approval": "never"
    }]
  }
}
```

**server\_label** `string` *required*

Label on the server's `mcp_list_tools` and `mcp_call` items.

**server\_url** `string` *required*

HTTPS server URL on a public host.

**authorization** `string`

Sent to the server as a Bearer token.

**headers** `object`

Other HTTP headers sent to the server.

**allowed\_tools** `string[]`

Only expose these tools.

**require\_approval** `string | object` (default: never)

`never`, `always`, or a filter such as `{"never": {"tool_names": ["search_docs"]}}`. See [Approvals](#approvals).

**x\_transport** `string` (default: streamable\_http)

`sse` for an SSE server.

**x\_call\_timeout\_secs** `float` (default: 30)

Per-call timeout in seconds.

The server reports discovery as an `mcp_list_tools` item and each call as an `mcp_call` item.

#### Approvals

With `require_approval`, a call waits for the client. The response ends with an `mcp_approval_request` item:

```json theme={"system"}
{ "type": "mcp_approval_request", "id": "item_abc", "name": "delete_doc",
  "arguments": "{\"id\": 7}", "server_label": "docs" }
```

Answer with an `mcp_approval_response` item, then send `response.create`:

```json theme={"system"}
{
  "type": "conversation.item.create",
  "item": { "type": "mcp_approval_response", "approval_request_id": "item_abc", "approve": true }
}
```

The server doesn't run a rejected request, or one left unanswered for 5 minutes.

#### Delegation

Delegation hands over the session's tool calls and decisions to a stronger model on [Model APIs](https://docs.baseten.co/inference/model-apis/overview.md) while the voice model keeps the conversation. It uses your Baseten API key.

```json theme={"system"}
{ "type": "session.update", "session": { "type": "realtime",
  "x_delegation": { "model": "zai-org/GLM-5.3", "api_key": "<BASETEN_API_KEY>" } } }
```

`false` turns it off, and `true` turns it back on with its last settings. `session.updated` reports the settings in `x_providers.delegation`, without the key.

**x\_delegation.api\_key** `string`

Your Baseten API key. Required to enable delegation.

**x\_delegation.model** `string`

Model APIs model. Required the first time delegation is turned on.

**x\_delegation.max\_tokens** `integer`

Token cap for each delegate reply.

**x\_delegation.reasoning\_effort** `string` (default: none)

`none`, `low`, `medium`, or `high`. Any other value rejects the update. Unlike `x_llm.reasoning_effort`, it accepts `none`.

While delegation is enabled, the delegate's tool calls arrive as `function_call` items; reply with `function_call_output`, without `response.create`. The server reports handoffs as `mcp_call` items with `server_label: "delegation"`, and the delegate's steps with `server_label: "backend"`.

When the delegate's last spoken line asks the caller a question, the caller's next spoken turn goes straight to the delegate, without the voice model. This continues for as long as each of its replies ends with a question, so a delegate that closes every answer with "Is there anything else?" answers the rest of the call. See [Delegation](https://docs.baseten.co/inference/audio/realtime-voice/prompting-guide.md#delegation) in the prompting guide.

The `delegation` field uses the Responses API format, with `responses.model`, `responses.reasoning.effort`, and `responses.max_output_tokens`. If you send both fields, `x_delegation` takes precedence.

#### Use your own tools and keys

One `session.update` can combine client tools, an MCP server, and an LLM endpoint, using separate keys for the server and the LLM:

```json theme={"system"}
{
  "type": "session.update",
  "session": {
    "instructions": "You are a support agent. Look up accounts before answering.",
    "tools": [{
      "type": "function",
      "name": "lookup_account",
      "description": "Look up an account by email address.",
      "parameters": {
        "type": "object",
        "properties": {"email": {"type": "string"}},
        "required": ["email"]
      }
    }, {
      "type": "mcp",
      "server_label": "tickets",
      "server_url": "https://mcp.example.com/mcp",
      "authorization": "<MCP_API_KEY>",
      "allowed_tools": ["search_tickets"]
    }],
    "x_llm": {
      "base_url": "https://inference.baseten.co/v1",
      "model": "zai-org/GLM-5.3-Fast",
      "api_key": "<BASETEN_API_KEY>"
    }
  }
}
```

Keys sent this way apply to the session only. To switch back to the Chain's own LLM, send `{"provider": "internal"}`.

### input\_audio\_buffer.append

**audio** `string` *required*

Base64-encoded audio in the format set by `audio.input.format`.

### input\_audio\_buffer.commit

In manual mode, ends the caller's turn. The server replies with `input_audio_buffer.committed` and the turn's transcript events. A commit with less than 100 ms of new audio returns an `error` with code `input_audio_buffer_commit_empty`.

### input\_audio\_buffer.clear

Clears buffered input audio. The server replies with `input_audio_buffer.cleared`.

### output\_audio\_buffer.clear

Stops the agent's audio, as a barge-in does.

### conversation.item.create

Adds an item to the conversation. The server replies with `conversation.item.created`, or an `error` with code `item_not_found` when no call or approval is waiting for the ID.

**item.type** `string` *required*

`message`, `function_call_output`, or `mcp_approval_response`.

**item.role** `string` (default: user)

For `message` items: `user`, `assistant`, or `system`.

**item.content** `array`

For `message` items: `input_text` or `input_audio` parts for `user`, `input_text` for `system`, and `output_text` for `assistant`.

**item.call\_id** `string`

For `function_call_output` items: the call's `call_id`.

**item.output** `string | object`

For `function_call_output` items: the tool result.

**item.approval\_request\_id** `string`

For `mcp_approval_response` items: the request's `id`, with `approve` (`boolean`) and an optional `reason`.

### conversation.item.retrieve

**item\_id** `string` *required*

ID of the item to retrieve.

The server replies with `conversation.item.retrieved`. Audio isn't kept, so audio parts carry only their transcripts.

### response.create

Asks the agent to respond. A request sent during a response runs after it.

**response.instructions** `string`

Extra instructions for this response.

**response.input** `array`

`message` items, or `{"type": "item_reference", "id": "<item id>"}`, to add before responding.

**response.tool\_choice** `string | object`

Tool choice for this response.

**response.max\_output\_tokens** `integer | string`

Token cap for this response.

**response.output\_modalities** `string[]`

`["text"]` or `["audio"]`.

**response.conversation** `string` (default: auto)

`auto` adds the response to the conversation. `none` runs it out of band, without tools.

**response.metadata** `object`

The server echoes this on the response.

### response.cancel

Stops the in-progress response and its audio. With no response in progress, the server replies with an `error` with code `response_cancel_not_active`.

### conversation.item.truncate

Trims an assistant item to the audio the caller heard and cancels the in-progress response. The server replies with `conversation.item.truncated`. Unlike OpenAI Realtime, you don't need a separate `response.cancel`.

**item\_id** `string` *required*

ID of the assistant item being played.

**content\_index** `integer` (default: 0)

Always `0`.

**audio\_end\_ms** `integer`

Milliseconds of the item's audio the caller heard.

### conversation.item.delete

Removes an item from the conversation. Deleting a `function_call` or `mcp_call` item also removes its result. The server replies with `conversation.item.deleted`.

**item\_id** `string` *required*

ID of the item to delete.

## Server events

### Session

| Event | Sent when |
| - | - |
| `session.created` | The connection opens. |
| `session.updated` | A `session.update` applies. |

The `session` object's `x_providers` block reports the effective configuration:

**x\_providers.llm** `object`

Provider, model, `server_tools`, the `default` provider, and the `available` providers.

**x\_providers.tts** `object`

Provider, the current `voice`, and `voices`: every name `audio.output.voice` accepts, or `null` when the TTS model can't list them. Read it from `session.created` to see the available voices before you pick one.

**x\_providers.stt** `object`

Route, `available` routes, `supports_context`, `tool_keywords`, and `learned_keywords`.

**x\_providers.client\_tools** `string[]`

Registered client tool names.

**x\_providers.mcp\_tools** `string[]`

Registered MCP tool names.

**x\_providers.delegation** `object | null`

[Delegation](#delegation) settings, or `null` when it's off.

The `session` object also carries `x_server_version`:

**x\_server\_version** `string`

The voice agent release the deployment runs, such as `0.1.0`, or `unknown` when it isn't recorded. Read-only; include it when you report an issue.

### Input audio

| Event | Sent when |
| - | - |
| `input_audio_buffer.speech_started` | The caller starts speaking. |
| `input_audio_buffer.speech_stopped` | The caller stops speaking. |
| `input_audio_buffer.committed` | A turn's audio is committed. |
| `input_audio_buffer.cleared` | In reply to `input_audio_buffer.clear`. |

### User transcription

| Event | Sent when |
| - | - |
| `conversation.item.created`, `conversation.item.added` | A user audio item starts. The server sends both for each item: handle one. |
| `conversation.item.input_audio_transcription.delta` | Part of the transcript is ready. |
| `conversation.item.input_audio_transcription.completed` | The final transcript is ready. |
| `conversation.item.input_audio_transcription.failed` | Transcription fails. |
| `conversation.item.done` | The item is final, once per item, with its transcript. |

### Conversation

| Event | Sent when |
| - | - |
| `conversation.item.created` | An item you added is in the conversation. |
| `conversation.item.done` | An `mcp_list_tools` item finishes, or an `mcp_call` that outlasted its response finishes. |
| `conversation.item.retrieved` | In reply to `conversation.item.retrieve`. |
| `conversation.item.truncated` | In reply to `conversation.item.truncate`. |
| `conversation.item.deleted` | In reply to `conversation.item.delete`. |

### Response

The server sends these events for a spoken response, in order:

1. `response.created`
2. `response.output_item.added`, then `response.content_part.added`
3. `response.output_audio_transcript.delta` and `response.output_audio.delta`, interleaved
4. `response.output_audio_transcript.done`, then `response.output_audio.done`
5. `response.content_part.done`, then `response.output_item.done`
6. `response.done`

For a text-only response, the server sends `response.output_text.delta` and `response.output_text.done` instead of the audio and transcript events.

`response.done` carries a `status` of `completed`, `cancelled`, or `incomplete`. For a cancelled response, `status_details.reason` is `turn_detected` when the caller interrupted, or `client_cancelled`. A response that hit its token cap is `incomplete` with reason `max_output_tokens`.

### Function calls

| Event | Sent when |
| - | - |
| `response.function_call_arguments.delta` | Part of a tool call's arguments is ready. |
| `response.function_call_arguments.done` | A tool call is ready. Reply with a `function_call_output` item. |

### Tools the agent runs

The server reports MCP tools, server-side tools, and [delegation](#delegation) as `mcp_call` items, which need no reply:

| Event | Sent when |
| - | - |
| `response.output_item.added` | The call starts. |
| `response.mcp_call_arguments.done` | Its arguments are final. |
| `response.mcp_call.in_progress` | The call is running. |
| `response.mcp_call.completed`, `response.mcp_call.failed` | It finishes. |
| `response.output_item.done` | The item is final, with its `output` or `error`. |

An MCP call can outlast the response that made it. That response ends (`response.done`) with the item still in progress, and the call finishes later under the same item ID: `response.mcp_call.completed` or `response.mcp_call.failed`, then `conversation.item.done` with the item's `output` or `error`. Read the result from whichever arrives, not from `response.done`. The agent then answers with the result in a new response.

`server_label` names the MCP server, the server-side tool's provider, `delegation`, or `backend`.

### Errors

`error` reports a problem with an event you sent, or a failed model call during a turn. It echoes the client's `event_id`, and `param` names the field at fault when there is one. A `session.update` that fails applies none of its changes.

| `code` | `type` | Sent when |
| - | - | - |
| `invalid_json` | `invalid_request_error` | A message isn't a JSON object. |
| `missing_required_parameter` | `invalid_request_error` | A message has no `type`. |
| `invalid_value` | `invalid_request_error` | An unknown event type, invalid base64 audio, or an invalid field, such as an unsupported audio format, an unknown voice, or a bad `session.type`. |
| `unsupported_event` | `invalid_request_error` | `response.create` in a transcription session. |
| `input_audio_buffer_commit_empty` | `invalid_request_error` | `input_audio_buffer.commit` with less than 100 ms of new audio. |
| `response_cancel_not_active` | `invalid_request_error` | `response.cancel` with no response in progress. |
| `item_not_found` | `invalid_request_error` | A `function_call_output` or `mcp_approval_response` that nothing is waiting for, or an item ID that doesn't exist. |
| `rate_limit_exceeded` | `server_error` | A model returned HTTP 429. |
| `null` | `server_error` | Any other failure in a model call or on the server. |

The server reports a failed transcription as `conversation.item.input_audio_transcription.failed`, not as `error`.

The server closes the WebSocket with `1013` when the models aren't ready in time or the deployment is at capacity: reconnect with backoff. It closes with `1011` when the models fail to start or a session fails on the server.

### Latency profile

`response.done` and the first `response.output_audio.delta` of each turn carry `x_profile` timings in seconds:

**x\_profile.stt\_s** `float`

End of speech to final transcript.

**x\_profile.turn\_s** `float`

Final transcript to LLM start.

**x\_profile.llm\_ttft\_s** `float | null`

LLM start to first token. `null` when the turn used a speculative reply: the agent starts writing a reply when the caller pauses and sends it if the turn ends there, so the reply has no separate first-token time.

**x\_profile.tts\_ttfb\_s** `float`

First token to first audio.

**x\_profile.e2e\_s** `float`

End of speech to first audio.

Other `x_profile` fields are for diagnostics only and can change between releases.

```python Python theme={"system"}
import asyncio
import base64
import json
import os

import websockets

chain_id = ""  # The Chain ID Baseten provided
url = f"wss://chain-{chain_id}.api.baseten.co/environments/production/websocket"
headers = {"Authorization": f"Bearer {os.environ['BASETEN_API_KEY']}"}


async def main():
    async with websockets.connect(url, additional_headers=headers) as ws:
        print(json.loads(await ws.recv())["type"])  # session.created

        await ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "instructions": "You are a concise voice assistant.",
                "audio": {
                    "input": {"format": {"type": "audio/pcm", "rate": 24000}},
                    "output": {"format": {"type": "audio/pcm", "rate": 24000}},
                },
                "tools": [{
                    "type": "function",
                    "name": "lookup_account",
                    "description": "Look up an account by email address.",
                    "parameters": {
                        "type": "object",
                        "properties": {"email": {"type": "string"}},
                        "required": ["email"],
                    },
                }],
            },
        }))

        # Start a turn with text so the example runs without a microphone.
        # For speech, stream 24 kHz PCM16 mono audio instead; server_vad ends
        # the turn and responds on its own:
        # await ws.send(json.dumps({
        #     "type": "input_audio_buffer.append",
        #     "audio": base64.b64encode(chunk).decode(),
        # }))
        await ws.send(json.dumps({
            "type": "conversation.item.create",
            "item": {
                "type": "message",
                "role": "user",
                "content": [{
                    "type": "input_text",
                    "text": "What plan is jane@example.com on?",
                }],
            },
        }))
        await ws.send(json.dumps({"type": "response.create"}))

        async for raw in ws:
            event = json.loads(raw)
            if event["type"] == "error":
                print("error:", event["error"])
            elif event["type"] == "response.output_audio.delta":
                audio = base64.b64decode(event["delta"])  # play this
            elif event["type"] == "response.output_audio_transcript.delta":
                print(event["delta"], end="", flush=True)
            elif event["type"] == "response.done":
                # A tool-call response is followed by another after the
                # tool result; stop when a response has no function calls.
                output = event["response"]["output"]
                if not any(item["type"] == "function_call" for item in output):
                    print()
                    break
            elif event["type"] == "response.function_call_arguments.done":
                await ws.send(json.dumps({
                    "type": "conversation.item.create",
                    "item": {
                        "type": "function_call_output",
                        "call_id": event["call_id"],
                        "output": {"plan": "enterprise"},
                    },
                }))
                await ws.send(json.dumps({"type": "response.create"}))


asyncio.run(main())
```

```json session.updated theme={"system"}
{
  "event_id": "event_abc123",
  "type": "session.updated",
  "session": {
    "id": "sess_abc123",
    "object": "realtime.session",
    "type": "realtime",
    "model": "google/gemma-4-31B-it",
    "output_modalities": ["audio"],
    "instructions": "You are a concise voice assistant.",
    "voice": "ryan",
    "audio": {
      "input": {
        "format": { "type": "audio/pcm", "rate": 24000 },
        "turn_detection": {
          "type": "server_vad",
          "threshold": 0.5,
          "prefix_padding_ms": 300,
          "silence_duration_ms": 400,
          "create_response": true,
          "interrupt_response": true
        },
        "transcription": { "model": "internal", "language": "English" }
      },
      "output": {
        "format": { "type": "audio/pcm", "rate": 24000 },
        "voice": "ryan",
        "speed": 1.0
      }
    },
    "tools": [{
      "type": "function",
      "name": "lookup_account",
      "description": "Look up an account by email address.",
      "parameters": {
        "type": "object",
        "properties": { "email": { "type": "string" } },
        "required": ["email"]
      }
    }],
    "tool_choice": "auto",
    "max_output_tokens": "inf",
    "x_providers": {
      "llm": { "provider": "internal", "default": "internal", "available": ["internal"], "model": "google/gemma-4-31B-it", "server_tools": [] },
      "tts": { "provider": "internal", "voice": "ryan", "voices": ["olivia", "ryan", "susan", "tyler"], "model": "Base" },
      "stt": {
        "provider": "internal",
        "available": ["internal"],
        "language": "English",
        "supports_context": true,
        "tool_keywords": true,
        "learned_keywords": []
      },
      "client_tools": ["lookup_account"],
      "mcp_tools": []
    },
    "x_server_version": "0.1.0"
  }
}
```
