Skip to main content
Baseten Realtime Voice is in Private Preview. Join the waitlist to get it enabled for your organization.
These practices apply to any client of Baseten Realtime Voice, whether you use a browser app, a phone system bridge, or the OpenAI Agents SDK. The API reference documents each event and field mentioned here.

Start a call

  • Configure before the caller speaks. Send one session.update with instructions, tools, and audio formats as soon as session.created arrives, then start streaming audio. The server applies events in order, so you don’t need to wait for session.updated. Switch the speech-to-text (STT) route in that first update: the route can’t change once caller audio arrives.
  • Capture while you connect. Start the microphone or telephony stream while the WebSocket opens, buffer the samples, and flush them once the session is configured, so the caller’s first words aren’t lost.
  • Retry busy and cold starts. A connection closed with code 1013 means the deployment is warming up or a replica is full; reconnect with exponential backoff. A new connection is a new session, so resend your session.update.

Send audio

  • Use the caller’s native format. Send G.711 (audio/pcmu or audio/pcma) straight from a phone system rather than converting it, and 16-bit pulse-code modulation (PCM) at your capture rate from apps. The agent resamples internally, so don’t resample on the client.
  • Stream continuously in small chunks. Send audio as it’s captured, in 20 to 100 ms chunks, and keep sending while the agent talks: barge-in and the end-of-turn model depend on hearing the caller the whole time. Don’t mute the microphone during playback.
  • Cancel echo on speakerphones. If the agent’s voice leaks back into the microphone, it can hear itself and stop mid-reply. Turn on echo cancellation (in browsers, echoCancellation: true; in Python, see Cancel echo in a Python client), or use a headset.
  • Size the uplink for many calls. Each 24 kHz PCM stream is about 0.5 Mbit/s as base64, 16 kHz about 0.35 Mbit/s, and G.711 about 0.09 Mbit/s. A relay that carries dozens of calls from one host can run out of upload bandwidth before the deployment runs out of capacity.

Cancel echo in a Python client

LiveKit’s AudioProcessingModule runs WebRTC’s echo canceller on raw PCM. It takes 10 ms frames, so open the microphone and speaker streams with blocksize=FRAME. Give it each frame you play as the reference, then clean each microphone frame against it before you send it:
Once both streams are open, tell it how far the echo lags: apm.set_stream_delay_ms(int((mic_stream.latency + speaker_stream.latency) * 1000)). Use separate input and output streams rather than one duplex stream: on a laptop the microphone and speakers are separate devices, and a duplex stream across them can glitch.

Handle playback and interruptions

  • Play deltas as they arrive. Queue each response.output_audio.delta for playback immediately rather than waiting for the response to finish.
  • Buffer before playback. Start playing once the queue holds about 200 ms, and buffer again whenever the queue runs dry, so network jitter doesn’t become clicks. When the response’s audio ends, play what’s left. If an audio callback thread reads the queue, guard it with a lock.
  • Stop playback when a response is cancelled. Short acknowledgments like “mm-hmm” don’t interrupt the agent, so don’t flush playback on input_audio_buffer.speech_started. When a response.done arrives with status: "cancelled" and reason turn_detected, stop playback and drop any audio still queued.
  • Report what the caller heard. After a barge-in, send conversation.item.truncate with the audio_end_ms you actually played, so the transcript and the conversation history end where the caller stopped listening.

Handle tool calls

  • Answer tool calls promptly. Reply to each client tool call with a function_call_output within 30 seconds; the agent’s spoken acknowledgment covers a slow one, but the caller still waits.
  • Write tool descriptions for the model. Name each tool for what it does, say when to call it, and keep parameters few and typed. See Function calling.
  • Keep MCP credentials on your server. A Model Context Protocol (MCP) server’s key travels in the session.update that adds it, in authorization or headers, so whoever sends that update holds the key. Send it from your backend or relay, never from a browser or mobile app. The key applies to that session only, and the agent never echoes authorization or headers back, in session.updated or any other event.
  • Scope what an MCP key can do. The model decides when to call an MCP tool, from what the caller says, so a caller can steer it. Give the agent a key limited to the tools it needs, read-only where you can, and short-lived. List those tools in allowed_tools, and set require_approval for any tool that changes data.
  • Put the rules in instructions. With delegation enabled, the delegate reads your instructions as the policy for the call, so write refund limits, verification steps, and handoff rules there, not in tool descriptions.

Measure latency

A turn's server latency, e2e_s, from the end of the caller's speech to the agent's first audio, split into stt_s, turn_s, llm_ttft_s, and tts_ttfb_s. A turn's server latency, e2e_s, from the end of the caller's speech to the agent's first audio, split into stt_s, turn_s, llm_ttft_s, and tts_ttfb_s.
  • Monitor x_profile. response.done and the first response.output_audio.delta of each turn carry per-stage timings: stt_s, turn_s, llm_ttft_s, tts_ttfb_s, and e2e_s, from the end of speech to the first audio on the server. What the caller hears also includes the pause while the agent confirms the turn ended, plus your network latency and playback buffering.
  • Keep the client near the deployment. Host your relay or telephony bridge in the same region as the deployment; every round trip adds to each turn. Baseten shares your deployment’s region with you.