Baseten Realtime Voice is in Private Preview. Join the waitlist
to get it enabled for your organization.
Start a call
- Configure before the caller speaks. Send one
session.updatewith instructions, tools, and audio formats as soon assession.createdarrives, then start streaming audio. The server applies events in order, so you don’t need to wait forsession.updated. Switch the speech-to-text (STT) route in that first update: the route can’t change once caller audio arrives. - Capture while you connect. Start the microphone or telephony stream while the WebSocket opens, buffer the samples, and flush them once the session is configured, so the caller’s first words aren’t lost.
- Retry busy and cold starts. A connection closed with code
1013means the deployment is warming up or a replica is full; reconnect with exponential backoff. A new connection is a new session, so resend yoursession.update.
Send audio
- Use the caller’s native format. Send G.711 (
audio/pcmuoraudio/pcma) straight from a phone system rather than converting it, and 16-bit pulse-code modulation (PCM) at your capture rate from apps. The agent resamples internally, so don’t resample on the client. - Stream continuously in small chunks. Send audio as it’s captured, in 20 to 100 ms chunks, and keep sending while the agent talks: barge-in and the end-of-turn model depend on hearing the caller the whole time. Don’t mute the microphone during playback.
- Cancel echo on speakerphones. If the agent’s voice leaks back into the microphone, it can hear itself and stop mid-reply. Turn on echo cancellation (in browsers,
echoCancellation: true; in Python, see Cancel echo in a Python client), or use a headset. - Size the uplink for many calls. Each 24 kHz PCM stream is about 0.5 Mbit/s as base64, 16 kHz about 0.35 Mbit/s, and G.711 about 0.09 Mbit/s. A relay that carries dozens of calls from one host can run out of upload bandwidth before the deployment runs out of capacity.
Cancel echo in a Python client
LiveKit’sAudioProcessingModule runs WebRTC’s echo canceller on raw PCM. It takes 10 ms frames, so open the microphone and speaker streams with blocksize=FRAME. Give it each frame you play as the reference, then clean each microphone frame against it before you send it:
apm.set_stream_delay_ms(int((mic_stream.latency + speaker_stream.latency) * 1000)). Use separate input and output streams rather than one duplex stream: on a laptop the microphone and speakers are separate devices, and a duplex stream across them can glitch.
Handle playback and interruptions
- Play deltas as they arrive. Queue each
response.output_audio.deltafor playback immediately rather than waiting for the response to finish. - Buffer before playback. Start playing once the queue holds about 200 ms, and buffer again whenever the queue runs dry, so network jitter doesn’t become clicks. When the response’s audio ends, play what’s left. If an audio callback thread reads the queue, guard it with a lock.
- Stop playback when a response is cancelled. Short acknowledgments like “mm-hmm” don’t interrupt the agent, so don’t flush playback on
input_audio_buffer.speech_started. When aresponse.donearrives withstatus: "cancelled"and reasonturn_detected, stop playback and drop any audio still queued. - Report what the caller heard. After a barge-in, send
conversation.item.truncatewith theaudio_end_msyou actually played, so the transcript and the conversation history end where the caller stopped listening.
Handle tool calls
- Answer tool calls promptly. Reply to each client tool call with a
function_call_outputwithin 30 seconds; the agent’s spoken acknowledgment covers a slow one, but the caller still waits. - Write tool descriptions for the model. Name each tool for what it does, say when to call it, and keep parameters few and typed. See Function calling.
- Keep MCP credentials on your server. A Model Context Protocol (MCP) server’s key travels in the
session.updatethat adds it, inauthorizationorheaders, so whoever sends that update holds the key. Send it from your backend or relay, never from a browser or mobile app. The key applies to that session only, and the agent never echoesauthorizationorheadersback, insession.updatedor any other event. - Scope what an MCP key can do. The model decides when to call an MCP tool, from what the caller says, so a caller can steer it. Give the agent a key limited to the tools it needs, read-only where you can, and short-lived. List those tools in
allowed_tools, and setrequire_approvalfor any tool that changes data. - Put the rules in
instructions. With delegation enabled, the delegate reads yourinstructionsas the policy for the call, so write refund limits, verification steps, and handoff rules there, not in tool descriptions.
Measure latency
- Monitor
x_profile.response.doneand the firstresponse.output_audio.deltaof each turn carry per-stage timings:stt_s,turn_s,llm_ttft_s,tts_ttfb_s, ande2e_s, from the end of speech to the first audio on the server. What the caller hears also includes the pause while the agent confirms the turn ended, plus your network latency and playback buffering. - Keep the client near the deployment. Host your relay or telephony bridge in the same region as the deployment; every round trip adds to each turn. Baseten shares your deployment’s region with you.