Baseten Realtime Voice is in Private Preview. Join the waitlist
to get it enabled for your organization.
instructions string in session.update. It replaces the deployment’s default prompt and controls what the agent says. Other fields set how the agent sounds and hears: audio.output.voice picks the voice, and audio.input.transcription.language sets the language that speech-to-text (STT) expects.
This guide builds on the phone support agent recipe. The examples use the same company, Northwind Fiber.
What the agent adds to your instructions
Baseten Realtime Voice appends its own guidance to yourinstructions, so your prompt only needs to cover your business’s specifics:
- Spoken style. Short replies, one question at a time, no lists, Markdown, or emoji. The appended guidance also covers how to collect and look up names, email addresses, phone numbers, and codes, and asks the caller to agree before making a change. To write your own style section instead, set
spoken_style: false. - Tool acknowledgments. When the session has tools, the agent says a short line such as “Thanks, checking that now” when a call takes more than a moment. Don’t script your own preamble on top of it. To write your own, set
tool_acknowledgement: false. - Delegation. With delegation enabled, the delegate reads your
instructionsas the policy for the call, and the voice model reads them for how to speak.
Prompting tips
- Change one thing at a time. Small wording changes can shift behavior a lot. Test each change on a few calls before you make the next.
- Prefer short bullet lists to paragraphs. The model follows bullet lists more reliably than paragraphs.
- Give example phrases. The agent copies sample lines closely, so write them the way you want them said, and give a few so replies don’t repeat.
- Be precise. Ambiguous or conflicting rules give inconsistent calls. Say what to do in the unclear case.
- Emphasize sparingly.
ALWAYSandNEVERin capitals work for a few hard rules; on every line they stop meaning anything.
Recommended structure
Write the prompt in Markdown with these sections in this order, and write it in the second person (“you”):Role
Say who the agent is, who it works for, and that it’s on a call. One or two sentences are enough. A name gives the agent an answer when the caller asks who they’re speaking with.Objective
State the outcome the agent works toward and what it does when it can’t reach it:Conversation flow
Describe the call flow, naming only tools in the session. A few sentences of prose suit most agents, as in the earlier example. If the agent skips steps or stalls, break the call into phases, each with a goal and a condition for moving on:Facts
- Put short facts in the prompt. Prices, hours, and policies the agent quotes go in a
## Factssection, in the words you want it to use. - Look up the rest. Account data, order status, and anything long or changing belongs behind a client tool or Model Context Protocol (MCP) server.
- Say what to do without an answer. Tell the agent to say it doesn’t know rather than guess.
Guardrails and escalation
List what the agent won’t do and when it hands off: an explicit request for a human, a caller who stays upset, a tool that keeps failing, or a topic outside its scope. For a handoff that your system carries out, give the agent a client tool such astransfer_to_specialist, and write the line the agent says when it calls the tool:
Language and terms
- Set the reply language. Add a line such as “Respond only in English. If the caller speaks another language, say you can only help in English.” Set
audio.input.transcription.languageto match, so STT expects that language. - Help STT hear your names. List product and company names in
audio.input.transcription.keywords. The recipes show the format.
Tools
- Mention only tools the session has. A prompt that names a tool not in
toolsmakes the agent try to call it, or claim it did. - Describe when to call each tool. Keep the prompt for the order of steps; the tool description says what each tool does and when.
- Call read-only tools without asking. Lookups and status checks don’t need the caller’s permission. The agent already asks before it changes anything.
Delegation
When delegation is enabled, a stronger model reads yourinstructions as the policy for the call, calls your tools, and writes what the caller hears. Write the prompt as that policy:
- Write the rules. Put limits, verification steps, and handoff conditions in
instructions, such as “Credits over $50 need a supervisor; offer a callback instead.” - Include the policy text. Paste the billing or returns policy the delegate applies, so it decides from the policy rather than from what the caller says.
- End on a question only to get an answer. When the delegate’s line ends with a question, the caller’s reply goes straight back to it, without the voice model. Ask when the delegate needs the caller’s answer, such as an order number or a confirmation. A closing “Is there anything else?” after every answer keeps the rest of the call on the slower delegate.
- Don’t route the work yourself. Baseten Realtime Voice decides what to hand to the delegate, so the prompt doesn’t need rules about when to delegate.
Greet the caller
The greeting isn’t part ofinstructions. To have the agent speak first, send response.create with per-response instructions once the session is configured. Baseten Realtime Voice adds the per-response instructions to the session’s instructions, so the agent stays in character:
Iterate on a prompt
Read the transcripts of test calls:conversation.item.input_audio_transcription.completed carries what the caller said, and response.output_audio_transcript.done carries what the agent said. When a call goes wrong, ask a more capable text model to review the prompt against the transcript: