Server-side tool execution

Early access. Server-side tool execution is available as a rate limited preview for playground usage (25 RPM across all models). For scaled evaluations or high-volume production integration, contact Baseten support or use the support chat in the Baseten app.

Use this reference to select tools that Baseten executes server-side, configure the execution loop, and parse protocol-specific responses. Server-side tool execution works with Messages, Responses, and ChatCompletions. See Web search for runnable requests and outputs.

Request fields

`x-baseten-server-tools`

Include this header on every request that uses server-side tool execution:

x-baseten-server-tools: true

The header routes the request through the server-side tool execution loop. Without it, the model endpoint rejects the baseten__* selector as an unknown tool type.

`tools`

Select a tool for Baseten to execute server-side with an object that contains its type:

{"type": "baseten__<provider>__<tool>"}

Selectors use baseten__<provider>__<tool> in the request’s tools array. Extra fields on a selector return 400.

Tool catalog

Search tools discover pages. Fetch tools retrieve content from known URLs. Give the model both when it should discover sources and then read selected pages.

Provider	Capability	Selector	Timeout
Exa	Search	`baseten__exa__web_search_exa`	15 seconds
Exa	Advanced search	`baseten__exa__web_search_advanced_exa`	30 seconds
Exa	Fetch	`baseten__exa__web_fetch_exa`	15 seconds
Keenable	Search	`baseten__keenable__search_web_pages`	15 seconds
Keenable	Fetch	`baseten__keenable__fetch_page_content`	15 seconds
Parallel	Search	`baseten__parallel__web_search`	30 seconds
Parallel	Fetch	`baseten__parallel__web_fetch`	30 seconds
You.com	Search	`baseten__youcom__you-search`	30 seconds
You.com	Fetch	`baseten__youcom__you-contents`	30 seconds

The advanced Exa search schema adds domain filters, subpage crawling, and a summary option to the basic search.

Server-side tools run on third-party search providers. By adding their tools, you agree to their respective terms: Exa, Keenable, Parallel, You.com.

The provider’s tool schema defines the arguments, and the model supplies their values.

Request configuration

Set request-level loop limits under baseten.tool_settings:

{
  "baseten": {
    "tool_settings": {
      "max_react_iterations": 6,
      "max_tool_calls_per_iteration": 4
    }
  }
}

Field	Default	Accepted values	Behavior
`max_react_iterations`	`12`	`2` through `20`	Limits model iterations. Baseten reserves the final iteration for an answer, so `N` iterations allow up to `N - 1` tool-call rounds. Baseten accepts `1` only when the request cannot call a server tool, for example when `tool_choice` disables tools.
`max_tool_calls_per_iteration`	`10`	`1` through `10`	Limits server-side tool calls in one model iteration. Calls over the limit become error results that the model can handle.

On Responses, the native max_tool_calls field sets a whole-request budget of server-side tool calls. When the loop spends the budget, Baseten steers the model to answer from the results it already has. Messages and ChatCompletions have no equivalent field.

tool_choice

Most requests can omit tool_choice and let the model decide whether to use a tool. Set it when the first model iteration must call a tool, call a particular tool, or avoid tools.tool_choice applies only to the first model iteration. Later iterations use automatic selection so the model can stop calling tools and answer.

Changing tool_choice between requests invalidates the KV cache. In a multi-turn conversation, keep tool_choice stable, or change it infrequently.

Intent	`Messages`	`Responses`	`ChatCompletions`
Let the model decide	Omit or `{"type":"auto"}`	Omit or `"auto"`	Omit or `"auto"`
Require any tool	`{"type":"any"}`	`"required"`	`"required"`
Require one tool	`{"type":"tool","name":"baseten__<provider>__<tool>"}`	`{"type":"function","name":"baseten__<provider>__<tool>"}`	`{"type":"function","function":{"name":"baseten__<provider>__<tool>"}}`
Disable tools	`{"type":"none"}`	`"none"`	`"none"`

If a request includes both server-side and application-side tool execution, the require any tool forms allow the model to choose either type. Name the server-side tool when the first iteration must search.

Response shapes

Each protocol returns the final answer through its normal response field and represents completed server-side calls differently. A turn that ends on an open application call reports the protocol’s tool-calling stop condition. Messages reports stop_reason: "tool_use", Responses has no stop field and carries the open call as a function_call item in output, and ChatCompletions reports finish_reason: "tool_calls".

Protocol	Final answer	Completed server-side calls	Open application calls
`Messages`	`content` text blocks	`tool_use` and `tool_result` blocks	`tool_use` blocks without a matching result
`Responses`	`output` message items, exposed as `output_text` by the OpenAI SDK	Completed `mcp_call` items	`function_call` items
`ChatCompletions`	`choices[0].message.content`	`baseten.iterations[].continuation_messages`	`choices[0].message.tool_calls`

Messages uses generic tool_use and tool_result blocks for tools that Baseten executes server-side. These blocks round-trip through Baseten’s Messages endpoint, but Anthropic’s API does not accept tool_result inside an assistant message. The top-level usage is the billed total across all model iterations, and usage.input_tokens excludes cache reads. ChatCompletions concatenates the text of every iteration into choices[0].message.content. A preamble such as “I will search now” arrives attached to the answer with no separator. Read baseten.iterations[].continuation_messages for the separated transcript. Responses uses completed mcp_call items for tools that Baseten executes server-side. server_label identifies the provider. Application tools remain standard function_call items.

`baseten` response metadata

The optional baseten object describes work across model iterations. Read it to display ChatCompletions tool execution progress, retain ChatCompletions search history, inspect per-iteration usage, or distinguish an iteration limit from a model token limit. A client that only displays the final answer can ignore it. The object has two scopes. iterations[] holds one entry per internal model call. request holds fields that describe the whole HTTP request.

Extension example (ChatCompletions, buffered)

"baseten": {
  "iterations": [
    {
      "index": 0,
      "usage": {"prompt_tokens": 599, "completion_tokens": 414, "total_tokens": 1013},
      "server_tool_calls": [
        {
          "id": "baseten__<provider>__<tool>_0_abcd1234",
          "provider": "<provider>",
          "name": "baseten__<provider>__<tool>",
          "arguments": "{\"query\": \"…\"}",
          "is_error": false
        }
      ],
      "continuation_messages": [
        {"role": "assistant", "tool_calls": ["…"]},
        {"role": "tool", "tool_call_id": "…", "content": "{\"results\": […]}"}
      ],
      "debug_msg": ["appended steering message: …"]
    }
  ],
  "request": {
    "termination_reason": "max_react_iterations_reached",
    "server_tool_calls": [
      {
        "id": "baseten__<provider>__<tool>_0_abcd1234",
        "provider": "<provider>",
        "name": "baseten__<provider>__<tool>",
        "status": "succeeded",
        "billable": true,
        "sku": "search",
        "quantity": 1.0
      }
    ]
  }
}

Field	Protocol	Use it to
`iterations[].index`	All	Merge partial metadata from the same iteration while streaming.
`iterations[].usage`	All	Inspect token usage for one model iteration.
`iterations[].server_tool_calls[]`	`ChatCompletions`	Display a server-side call’s provider, name, arguments, completion state, and error state. Merge repeated records by `id`.
`iterations[].continuation_messages[]`	`ChatCompletions`	Append the complete server-side tool execution transcript to a follow-up request.
`iterations[].debug_msg[]`	All	Read disclosed server-side notes, such as the final-iteration steering message. Free-form strings. Do not branch on them.
`request.termination_reason`	All	Distinguish a server-side tool execution iteration limit from the protocol’s native stop condition.
`request.server_tool_calls[]`	All	Inspect the outcome of every server-side call in the request: `status`, `billable`, `sku`, `quantity`.

The billing fields billable, sku, and quantity are provisional during early access. Their names and semantics can change without notice. sku can be null, and the meaning of quantity varies by provider. Baseten may add fields to this object. Ignore fields that your client does not use.

Streaming metadata

Server-side tool execution supports buffered and streaming requests. The baseten object appears in these locations:

Protocol	Buffered	Streaming
`Messages`	Response body root	The in-flight events the SDK does not rebuild, such as `content_block_start` and `message_delta`. `message_start` carries no metadata.
`Responses`	Response body root	Event root on events that carry the extension, and inside `response` on the terminal event
`ChatCompletions`	Response body root	Chunks with an empty `choices` array

Streaming iteration entries are partial. Merge iterations by index and ChatCompletions server-side calls by id. Append each continuation_messages array once, in arrival order. ChatCompletions metadata chunks have choices: []. Check for an empty array before reading choices[0] in a hand-written streaming client.

Iteration limit

If the model continues to call server-side tools at the iteration limit, Baseten returns a resumable response with max_react_iterations_reached:

Protocol	Native status	Detailed reason
`Messages`	`stop_reason: "pause_turn"`	`baseten.request.termination_reason`
`Responses`	`status: "completed"` (unchanged)	`baseten.request.termination_reason`
`ChatCompletions`	`finish_reason: "length"`	`baseten.request.termination_reason`

The transcript ends with a completed server-side call and result. Send it back without a new user message to resume from that point, or retry the original request with a larger max_react_iterations value. In ChatCompletions, finish_reason: "length" without baseten.request.termination_reason means the model reached max_tokens. OpenAI SDK helpers can raise LengthFinishReasonError in either case. The completion remains available as error.completion.

Observability

On enabled workspaces, a Model API page’s Metrics tab graphs server-side tool calls: throughput, throughput per tool, errors by error code, and call latency.
Billed tool calls appear in workspace settings under Billing → Usage, in the Tool calls view, grouped by provider.
Per-request detail is in the baseten response metadata: per-iteration usage, call outcomes, and request.termination_reason.
The metrics export endpoint does not include server-side tool series. The usage and costs REST APIs report token counts only, with no tool-call dimension.
A failure after a streaming response starts arrives as an SSE error event, because the HTTP status is already 200. Streaming clients must handle that event.

Limits and errors

Limit or condition	Behavior
Invalid selector, setting, or field under `baseten`	Returns `400`.
Feature not enabled for the organization	Returns `403` with `server-side tools are not enabled for this organization`.
Feature not enabled for the model	Returns `400` with `server-side tools are not enabled for this endpoint`.
Request duration	Stops after 285 seconds.
Request body	Accepts up to 10 MiB. Larger bodies return `413`.
Tool result size	Results over 1 MiB fail as an error tool result that the model can inspect.
Provider timeout	Stops the tool after the timeout listed in the tool catalog.
Server-tools rate limit	Returns `429` with `server-tools rate limit exceeded for this organization`. The early-access limit is 25 requests per minute, shared across all models. To raise it, contact Baseten support or use the support chat in the Baseten app.
Selected tool unavailable	Returns `400` when the tool’s provider is not connected.
Model API rate limit	Returns `429` with Model API semantics, distinct from the server-tools limit.
Model-plane failure or overload	`500`, `502`, and `529` pass through from the model plane, and `503` means the tool-execution fleet is at capacity, not the search provider. Same semantics as Model APIs. Retry with exponential backoff.

Server-side tool execution implements a subset of each protocol. A request that uses a refused feature returns 400 instead of silently degrading.

Protocol	Refused features
`Responses`	`store: true`, `previous_response_id`, `conversation`, and `background: true`. The loop keeps no server-side state. Echo the previous response’s `output` as the next request’s `input` to continue a conversation.
`ChatCompletions`	`n > 1` (multiple answer choices per request). The loop produces one choice per model turn.
All	Fixed arguments on a `baseten__*` selector. Extra fields return `400`.
All	Async inference requests.

Two output deviations from the stock protocols:

Messages represents server-side calls as generic tool_use and tool_result blocks, not Anthropic’s server_tool_use and web_search_tool_result types. Thinking blocks carry signature: "erased". Echo these blocks back to Baseten only, not to Anthropic’s API.
Responses does not echo request parameters on the response envelope. Fields such as temperature, tools, and store are null on the response.

A provider failure usually becomes a tool result that the model can inspect. The model can retry, change its query, choose another offered tool, or answer without that result. Errors that prevent the loop from starting return a non-2xx response. For general inference error guidance, see Inference errors. Use a client read timeout longer than 285 seconds for buffered requests. Streaming responses send a : keep-alive comment line every 15 seconds while a server-side call runs.

Reference

Inference API

Management API

CLI reference

SDK reference

Loops API

Training API

Frontier Gateway API

CI/CD

Server-side tool execution

Request fields

`x-baseten-server-tools`

`tools`

Tool catalog

Request configuration

Response shapes

`baseten` response metadata

Streaming metadata

Iteration limit

Observability

Limits and errors

​Request fields

​x-baseten-server-tools

​tools

​Tool catalog

​Request configuration

​Response shapes

​baseten response metadata

​Streaming metadata

​Iteration limit

​Observability

​Limits and errors

Request fields

`x-baseten-server-tools`

`tools`

Tool catalog

Request configuration

Response shapes

`baseten` response metadata

Streaming metadata

Iteration limit

Observability

Limits and errors