Skip to main content
Baseten Realtime Voice is in Private Preview. Join the waitlist to get it enabled for your organization.
The system prompt is the instructions string in session.update. It replaces the deployment’s default prompt and controls what the agent says. Other fields set how the agent sounds and hears: audio.output.voice picks the voice, and audio.input.transcription.language sets the language that speech-to-text (STT) expects. This guide builds on the phone support agent recipe. The examples use the same company, Northwind Fiber.

What the agent adds to your instructions

Baseten Realtime Voice appends its own guidance to your instructions, so your prompt only needs to cover your business’s specifics:
  • Spoken style. Short replies, one question at a time, no lists, Markdown, or emoji. The appended guidance also covers how to collect and look up names, email addresses, phone numbers, and codes, and asks the caller to agree before making a change. To write your own style section instead, set spoken_style: false.
  • Tool acknowledgments. When the session has tools, the agent says a short line such as “Thanks, checking that now” when a call takes more than a moment. Don’t script your own preamble on top of it. To write your own, set tool_acknowledgement: false.
  • Delegation. With delegation enabled, the delegate reads your instructions as the policy for the call, and the voice model reads them for how to speak.
Restating these rules in your prompt costs tokens on every turn, and a conflicting version of one makes the agent less predictable.

Prompting tips

  • Change one thing at a time. Small wording changes can shift behavior a lot. Test each change on a few calls before you make the next.
  • Prefer short bullet lists to paragraphs. The model follows bullet lists more reliably than paragraphs.
  • Give example phrases. The agent copies sample lines closely, so write them the way you want them said, and give a few so replies don’t repeat.
  • Be precise. Ambiguous or conflicting rules give inconsistent calls. Say what to do in the unclear case.
  • Emphasize sparingly. ALWAYS and NEVER in capitals work for a few hard rules; on every line they stop meaning anything.
Write the prompt in Markdown with these sections in this order, and write it in the second person (“you”):

Role

Say who the agent is, who it works for, and that it’s on a call. One or two sentences are enough. A name gives the agent an answer when the caller asks who they’re speaking with.

Objective

State the outcome the agent works toward and what it does when it can’t reach it:

Conversation flow

Describe the call flow, naming only tools in the session. A few sentences of prose suit most agents, as in the earlier example. If the agent skips steps or stalls, break the call into phases, each with a goal and a condition for moving on:

Facts

  • Put short facts in the prompt. Prices, hours, and policies the agent quotes go in a ## Facts section, in the words you want it to use.
  • Look up the rest. Account data, order status, and anything long or changing belongs behind a client tool or Model Context Protocol (MCP) server.
  • Say what to do without an answer. Tell the agent to say it doesn’t know rather than guess.

Guardrails and escalation

List what the agent won’t do and when it hands off: an explicit request for a human, a caller who stays upset, a tool that keeps failing, or a topic outside its scope. For a handoff that your system carries out, give the agent a client tool such as transfer_to_specialist, and write the line the agent says when it calls the tool:
An agent that may talk to someone in distress needs an escalation path like the second rule.

Language and terms

  • Set the reply language. Add a line such as “Respond only in English. If the caller speaks another language, say you can only help in English.” Set audio.input.transcription.language to match, so STT expects that language.
  • Help STT hear your names. List product and company names in audio.input.transcription.keywords. The recipes show the format.

Tools

  • Mention only tools the session has. A prompt that names a tool not in tools makes the agent try to call it, or claim it did.
  • Describe when to call each tool. Keep the prompt for the order of steps; the tool description says what each tool does and when.
  • Call read-only tools without asking. Lookups and status checks don’t need the caller’s permission. The agent already asks before it changes anything.

Delegation

When delegation is enabled, a stronger model reads your instructions as the policy for the call, calls your tools, and writes what the caller hears. Write the prompt as that policy:
  • Write the rules. Put limits, verification steps, and handoff conditions in instructions, such as “Credits over $50 need a supervisor; offer a callback instead.”
  • Include the policy text. Paste the billing or returns policy the delegate applies, so it decides from the policy rather than from what the caller says.
  • End on a question only to get an answer. When the delegate’s line ends with a question, the caller’s reply goes straight back to it, without the voice model. Ask when the delegate needs the caller’s answer, such as an order number or a confirmation. A closing “Is there anything else?” after every answer keeps the rest of the call on the slower delegate.
  • Don’t route the work yourself. Baseten Realtime Voice decides what to hand to the delegate, so the prompt doesn’t need rules about when to delegate.
The delegation recipe shows a complete session.

Greet the caller

The greeting isn’t part of instructions. To have the agent speak first, send response.create with per-response instructions once the session is configured. Baseten Realtime Voice adds the per-response instructions to the session’s instructions, so the agent stays in character:

Iterate on a prompt

Read the transcripts of test calls: conversation.item.input_audio_transcription.completed carries what the caller said, and response.output_audio_transcript.done carries what the agent said. When a call goes wrong, ask a more capable text model to review the prompt against the transcript:
Test the revision on the call that failed and on a few that worked, since a fix for one case can break another.

Starter template