> ## Documentation Index
> Fetch the complete documentation index at: https://docs.baseten.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Baseten Realtime Voice best practices

> Configure calls, stream audio, handle interruptions and tool calls, and measure latency with Baseten Realtime Voice.

> [!NOTE]
>
> Baseten Realtime Voice is in **Private Preview**. [Join the waitlist](https://baseten.co/talk-to-us/voice-beta-launch)
> to get it enabled for your organization.

These practices apply to any client of Baseten Realtime Voice, whether you use a browser app, a phone system bridge, or the OpenAI Agents SDK. The [API reference](https://docs.baseten.co/reference/inference-api/predict-endpoints/realtime-voice-api.md) documents each event and field mentioned here.

## Start a call

* **Configure before the caller speaks.** Send one `session.update` with instructions, tools, and audio formats as soon as `session.created` arrives, then start streaming audio. The server applies events in order, so you don't need to wait for `session.updated`. Switch the speech-to-text (STT) route in that first update: the route can't change once caller audio arrives.
* **Capture while you connect.** Start the microphone or telephony stream while the WebSocket opens, buffer the samples, and flush them once the session is configured, so the caller's first words aren't lost.
* **Retry busy and cold starts.** A connection closed with code `1013` means the deployment is warming up or a replica is full; reconnect with exponential backoff. A new connection is a new session, so resend your `session.update`.

## Send audio

* **Use the caller's native format.** Send G.711 (`audio/pcmu` or `audio/pcma`) straight from a phone system rather than converting it, and 16-bit pulse-code modulation (PCM) at your capture rate from apps. The agent resamples internally, so don't resample on the client.
* **Stream continuously in small chunks.** Send audio as it's captured, in 20 to 100 ms chunks, and keep sending while the agent talks: barge-in and the end-of-turn model depend on hearing the caller the whole time. Don't mute the microphone during playback.
* **Cancel echo on speakerphones.** If the agent's voice leaks back into the microphone, it can hear itself and stop mid-reply. Turn on echo cancellation (in browsers, `echoCancellation: true`; in Python, see [Cancel echo in a Python client](#cancel-echo-in-a-python-client)), or use a headset.
* **Size the uplink for many calls.** Each 24 kHz PCM stream is about 0.5 Mbit/s as base64, 16 kHz about 0.35 Mbit/s, and G.711 about 0.09 Mbit/s. A relay that carries dozens of calls from one host can run out of upload bandwidth before the deployment runs out of capacity.

### Cancel echo in a Python client

LiveKit's `AudioProcessingModule` runs WebRTC's echo canceller on raw PCM. It takes 10 ms frames, so open the microphone and speaker streams with `blocksize=FRAME`. Give it each frame you play as the reference, then clean each microphone frame against it before you send it:

```python theme={"system"}
from livekit import rtc  # pip install livekit

RATE, FRAME = 24_000, 240  # 10 ms at 24 kHz
apm = rtc.AudioProcessingModule(echo_cancellation=True, noise_suppression=True)


def on_speaker(outdata, frames, time, status):
    ...  # fill outdata from your playback queue
    apm.process_reverse_stream(rtc.AudioFrame(bytes(outdata), RATE, 1, FRAME))


def on_mic(indata, frames, time, status):
    mic = bytearray(indata)
    apm.process_stream(rtc.AudioFrame(mic, RATE, 1, FRAME))  # cancels the echo in place
    send_audio(bytes(mic))
```

Once both streams are open, tell it how far the echo lags: `apm.set_stream_delay_ms(int((mic_stream.latency + speaker_stream.latency) * 1000))`. Use separate input and output streams rather than one duplex stream: on a laptop the microphone and speakers are separate devices, and a duplex stream across them can glitch.

## Handle playback and interruptions

* **Play deltas as they arrive.** Queue each `response.output_audio.delta` for playback immediately rather than waiting for the response to finish.
* **Buffer before playback.** Start playing once the queue holds about 200 ms, and buffer again whenever the queue runs dry, so network jitter doesn't become clicks. When the response's audio ends, play what's left. If an audio callback thread reads the queue, guard it with a lock.
* **Stop playback when a response is cancelled.** Short acknowledgments like "mm-hmm" don't interrupt the agent, so don't flush playback on `input_audio_buffer.speech_started`. When a `response.done` arrives with `status: "cancelled"` and reason `turn_detected`, stop playback and drop any audio still queued.
* **Report what the caller heard.** After a barge-in, send `conversation.item.truncate` with the `audio_end_ms` you actually played, so the transcript and the conversation history end where the caller stopped listening.

## Handle tool calls

* **Answer tool calls promptly.** Reply to each client tool call with a `function_call_output` within 30 seconds; the agent's spoken acknowledgment covers a slow one, but the caller still waits.
* **Write tool descriptions for the model.** Name each tool for what it does, say when to call it, and keep parameters few and typed. See [Function calling](https://docs.baseten.co/inference/function-calling.md).
* **Keep MCP credentials on your server.** A Model Context Protocol (MCP) server's key travels in the `session.update` that adds it, in `authorization` or `headers`, so whoever sends that update holds the key. Send it from your backend or relay, never from a browser or mobile app. The key applies to that session only, and the agent never echoes `authorization` or `headers` back, in `session.updated` or any other event.
* **Scope what an MCP key can do.** The model decides when to call an MCP tool, from what the caller says, so a caller can steer it. Give the agent a key limited to the tools it needs, read-only where you can, and short-lived. List those tools in `allowed_tools`, and set `require_approval` for any tool that changes data.
* **Put the rules in `instructions`.** With [delegation](https://docs.baseten.co/reference/inference-api/predict-endpoints/realtime-voice-api.md#delegation) enabled, the delegate reads your `instructions` as the policy for the call, so write refund limits, verification steps, and handoff rules there, not in tool descriptions.

## Measure latency

![A turn's server latency, e2e\_s, from the end of the caller's speech to the agent's first audio, split into stt\_s, turn\_s, llm\_ttft\_s, and tts\_ttfb\_s.](https://mintcdn.com/baseten-preview/MVxrL4x5r4Be5Wat/_images/realtime-voice-turn-latency-light.svg?fit=max\&auto=format\&n=MVxrL4x5r4Be5Wat\&q=85\&s=53568d7df67d20be0f166d454ac0e0e3)

* **Monitor `x_profile`.** `response.done` and the first `response.output_audio.delta` of each turn carry per-stage timings: `stt_s`, `turn_s`, `llm_ttft_s`, `tts_ttfb_s`, and `e2e_s`, from the end of speech to the first audio on the server. What the caller hears also includes the pause while the agent confirms the turn ended, plus your network latency and playback buffering.
* **Keep the client near the deployment.** Host your relay or telephony bridge in the same region as the deployment; every round trip adds to each turn. Baseten shares your deployment's region with you.
