> ## Documentation Index
> Fetch the complete documentation index at: https://docs.baseten.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Baseten Realtime Voice

> Run real-time, full-duplex voice agents with Baseten Realtime Voice, behind a single OpenAI Realtime-compatible WebSocket.

> [!NOTE]
>
> Baseten Realtime Voice is in **Private Preview**. [Join the waitlist](https://baseten.co/talk-to-us/voice-beta-launch)
> to get it enabled for your organization.

Baseten Realtime Voice runs speech-to-text (STT), a large language model (LLM), and text-to-speech (TTS) as one [Chain](https://docs.baseten.co/development/chain/overview.md) behind a single streaming endpoint. It implements the [OpenAI Realtime](https://platform.openai.com/docs/api-reference/realtime) event protocol, so Realtime clients that accept a full WebSocket URL connect to it as they would to OpenAI.

![The caller connects over an OpenAI Realtime-compatible WebSocket to the Realtime Voice service in Baseten Cloud, which runs speech-to-text, a voice model, and text-to-speech in one region, with custom functions and remote tool servers as server-side tools. The voice model delegates hard tasks to a stronger model on Model APIs.](https://mintcdn.com/baseten-preview/MVxrL4x5r4Be5Wat/images/audio/voice-agent-stack.png?fit=max\&auto=format\&n=MVxrL4x5r4Be5Wat\&q=85\&s=c5dbf8fdaa5efd971d0d7deab633288f)

## What the agent can do

* **Choose each model.** Pick the STT, LLM, and TTS model independently. STT and TTS run inside the Chain; the LLM can be any model deployed on Baseten, a model hosted on [Model APIs](https://docs.baseten.co/inference/model-apis/overview.md), or your own OpenAI-compatible endpoint. A session can switch the LLM, voice, or STT route without a redeploy. To compare STT and TTS models, see the [Speech and audio overview](https://docs.baseten.co/inference/audio/overview.md).
* **Natural turn-taking.** Voice activity detection, an end-of-turn model, and speculative replies (the agent starts a reply when the caller pauses, and sends it if the turn ends there) work together so the agent neither cuts callers off mid-thought nor leaves long gaps. Callers can interrupt, short acknowledgments like "mm-hmm" don't cancel the reply, and push-to-talk is one setting away.
* **Tools.** Give the agent client tools, remote Model Context Protocol (MCP) servers it calls directly, or web search through Model APIs. With delegation, a stronger model takes over tool calls and decisions while the voice model keeps the conversation. Before a slow tool call, the agent says a short acknowledgment so the line doesn't go quiet. For guidance on tool definitions that the LLM can use reliably, see [Function calling](https://docs.baseten.co/inference/function-calling.md).

You control each capability with `session.update` fields, which the [API reference](https://docs.baseten.co/reference/inference-api/predict-endpoints/realtime-voice-api.md) documents.

## Connect with the OpenAI Agents SDK

The [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/realtime/guide/) speaks the OpenAI Realtime protocol. This example runs a support call from your terminal: the agent greets you, you ask about an order, it looks up the order with a tool, and it answers. The agent detects the end of each turn and replies on its own, and you can talk over it.

To run this example, you need your Chain ID, which Baseten provides when it enables Realtime Voice for your organization, plus a [Baseten API key](https://docs.baseten.co/organization/api-keys.md). Install the dependencies with `pip install openai-agents sounddevice`, set `BASETEN_API_KEY`, and replace `<chain_id>` in the URL with your Chain ID.

```python theme={"system"}
import asyncio
import os

import sounddevice as sd
from agents import function_tool
from agents.realtime import RealtimeAgent, RealtimeRunner
from agents.realtime.model_inputs import RealtimeModelSendRawMessage

RATE = 24_000  # 16-bit mono PCM at 24 kHz, the agent's default
ORDERS = {"1042": "shipped yesterday, arriving Thursday", "1043": "being packed"}


# 1. Give the agent a tool. The SDK runs function tools in your process and sends
#    the result back; if one takes a moment, the agent says a short acknowledgment.
@function_tool
def lookup_order(order_id: str) -> str:
    """Look up an order's shipping status by its order number."""
    return ORDERS.get(order_id, "no order with that number")


# 2. Describe the agent. Cover your business; the agent adds voice-call guidance,
#    such as short replies and one question at a time, on its own.
agent = RealtimeAgent(
    name="Riley",
    instructions=(
        "You are Riley, a support agent for Northwind Outfitters on a live phone call. "
        "Help callers check on orders: ask for the order number, then look it up."
    ),
    tools=[lookup_order],
)


async def main():
    # 3. Connect to your Chain's WebSocket with your Baseten API key.
    session = await RealtimeRunner(agent).run(model_config={
        "url": "wss://chain-<chain_id>.api.baseten.co/environments/production/websocket",
        "api_key": os.environ["BASETEN_API_KEY"],
    })
    loop = asyncio.get_running_loop()
    speaker = bytearray()  # agent audio waiting to be played

    # 4. Stream the microphone the whole time, even while the agent talks:
    #    barge-in and end-of-turn detection depend on hearing the caller.
    def on_mic(indata, frames, time, status):
        asyncio.run_coroutine_threadsafe(session.send_audio(bytes(indata)), loop)

    # 5. Play the agent's audio as it arrives. This plays straight from a shared
    #    buffer to stay short. A production client buffers about 200 ms before
    #    playing so network jitter doesn't click, locks the buffer the audio thread
    #    reads, and cancels echo (or uses headphones). See the best practices.
    def on_speaker(outdata, frames, time, status):
        chunk = speaker[: len(outdata)]
        del speaker[: len(chunk)]
        outdata[: len(chunk)] = chunk
        outdata[len(chunk):] = bytes(len(outdata) - len(chunk))

    async with session:
        # 6. Speak first: ask for a greeting. Per-response instructions add to the
        #    session's, so the agent stays in character.
        await session.model.send_event(RealtimeModelSendRawMessage(message={
            "type": "response.create",
            "other_data": {"response": {"instructions": "Greet the caller as Riley "
                "from Northwind Outfitters and ask how you can help. One sentence."}},
        }))

        with sd.RawInputStream(samplerate=RATE, channels=1, dtype="int16", callback=on_mic), \
             sd.RawOutputStream(samplerate=RATE, channels=1, dtype="int16", callback=on_speaker):
            print("On a call with Riley. Ctrl+C to hang up.")
            # 7. Handle what the session reports: audio, interruptions, tool calls,
            #    and the transcripts of both sides.
            async for event in session:
                if event.type == "audio":
                    speaker.extend(event.audio.data)
                elif event.type == "audio_interrupted":
                    speaker.clear()  # the caller spoke over the agent: drop the rest
                elif event.type == "tool_end":
                    print(f"  [{event.tool.name}({event.arguments}) -> {event.output}]")
                elif event.type == "raw_model_event" and event.data.type == "raw_server_event":
                    server = event.data.data  # the server's own Realtime events
                    if server["type"] == "conversation.item.input_audio_transcription.completed":
                        print("You:  ", server["transcript"])
                    elif server["type"] == "response.output_audio_transcript.done":
                        print("Riley:", server["transcript"])


asyncio.run(main())
```

A call looks like this:

```text theme={"system"}
On a call with Riley. Ctrl+C to hang up.
Riley: Hi, this is Riley from Northwind Outfitters. How can I help you today?
You:   Hi, where's my order? It's 1042.
  [lookup_order({"order_id": "1042"}) -> shipped yesterday, arriving Thursday]
Riley: Your order 1042 shipped yesterday and is arriving Thursday.
```

Transcripts come from the server's events, read through `raw_model_event`: `history_added` items from the SDK don't carry the agent's transcript.

* Use headphones. The example doesn't cancel echo, so on speakers the agent hears itself and stops mid-reply. To cancel it in Python, see [Cancel echo in a Python client](https://docs.baseten.co/inference/audio/realtime-voice/best-practices.md#cancel-echo-in-a-python-client).
* On macOS, allow your terminal to use the microphone when prompted. Until you do, the agent hears silence.
* For a development deployment, connect to `/development/websocket` in place of `/environments/production/websocket`. The [API reference](https://docs.baseten.co/reference/inference-api/predict-endpoints/realtime-voice-api.md) lists every URL form.
* To check the connection without a microphone, run the text-only example at the end of the [API reference](https://docs.baseten.co/reference/inference-api/predict-endpoints/realtime-voice-api.md).

> [!NOTE]
>
> The endpoint accepts WebSocket connections only. Any Realtime client that accepts a full WebSocket URL can connect. The `openai` Python SDK appends `/realtime` to its base URL, which the endpoint doesn't serve, so it can't connect directly. Browsers can't set the `Authorization` header. Connect them through your own backend, which holds the API key and relays the socket.

### Set session fields from the SDK

The SDK sets the voice through `model_settings`. Voice names depend on the deployment's TTS model. For available voices and transcription settings, see [Voice and transcription settings](https://docs.baseten.co/inference/audio/realtime-voice/recipes.md#voice-and-transcription-settings).

```python theme={"system"}
runner = RealtimeRunner(agent, config={"model_settings": {"voice": "<voice name>"}})
```

`RealtimeAgent` accepts only function tools and rejects MCP tools. For [MCP tools](https://docs.baseten.co/inference/audio/realtime-voice/recipes.md#mcp-server), `x_`-prefixed fields, and other fields the SDK doesn't model, send a `session.update` as a raw event once the session is open, before you stream audio:

```python theme={"system"}
from agents.realtime.model_inputs import RealtimeModelSendRawMessage

async with session:
    await session.model.send_event(RealtimeModelSendRawMessage(message={
        "type": "session.update",
        "other_data": {"session": {"type": "realtime", "tools": [{
            "type": "mcp",
            "server_label": "deepwiki",
            "server_url": "https://mcp.deepwiki.com/mcp",
            "allowed_tools": ["ask_wiki_question"],
            "require_approval": "never",
        }]}},
    }))
```

The SDK merges `other_data` into the event body and checks the result against OpenAI's event schema. Keep `"type": "realtime"` in `session`: without it the check fails, and the SDK drops the event and logs `Failed to convert raw message`. Baseten's own fields, such as `tool_acknowledgement` or `x_llm`, pass the check. An update changes only the fields it sends, so it keeps the settings the SDK sent when it connected.

## Next steps

* Work through the [recipes](https://docs.baseten.co/inference/audio/realtime-voice/recipes.md), from a phone support agent to client tools, MCP servers, delegation, and switching the LLM per turn.
* Write your `instructions` with the [prompting guide](https://docs.baseten.co/inference/audio/realtime-voice/prompting-guide.md).
* Follow the [best practices](https://docs.baseten.co/inference/audio/realtime-voice/best-practices.md) for audio, interruptions, tool calls, and latency.
* Refer to the [API reference](https://docs.baseten.co/reference/inference-api/predict-endpoints/realtime-voice-api.md) for every event and `session.update` field.
* [Contact us](mailto:support@baseten.co) to get Baseten Realtime Voice enabled for your organization or to discuss your use case.
