Skip to main content
On Model APIs, deferred tool loading keeps tool schemas out of the prompt until the model asks for them. On Chat Completions and Messages, you mark tools with defer_loading: true and return tool_reference blocks for the matches. On Responses, you declare a client-side tool_search tool and return the matching definitions in tool_search_output. The model starts with the tools whose schemas are loaded and gets each deferred schema only when it asks for it. Use it when a request carries more tools than the model needs at once, such as an agent connected to several MCP servers or a shared internal tool catalog. Deferring unused schemas lowers input tokens and lets the model work with a catalog that would consume too much of the context window if you sent every schema up front. Baseten handles the schema work: the API keeps deferred schemas out of the prompt, loads a schema when the model asks for it, and keeps earlier turns unchanged for prefix caching. You provide the search tool and decide which tools it can return.

​
How deferred loading works

Deferred schemas don’t count as input tokens until the model asks for them. On Chat Completions and Messages, when the model references a tool, the server loads that tool’s schema. A referenced schema counts once. On Responses, the definitions in tool_search_output are the schemas the server loads for the next turn.

​
Model support

The table lists the models tested for deferred tool loading. Models that aren’t listed haven’t been verified; don’t send tool_reference blocks or tool_search_output definitions for them unless you confirm support. GLM 5.3 and GLM 5.3 Flash support deferred tool loading on all three surfaces. Kimi K3 and the DeepSeek V4.1 Flash models support it on Chat Completions and Messages. On Responses, they call tool_search, but the definitions in tool_search_output don’t become callable tools, so the model often doesn’t call them. GLM 5.3 Fast supports it on Chat Completions only. On Messages, defer_loading removes its schema but a tool_reference doesn’t restore it; GLM 5.3 Fast doesn’t call tool_search on Responses. GLM 5.2, Kimi K3.1 Pro, and DeepSeek V4 Pro 0813 ignore defer_loading on Chat Completions and Messages but support tool_search on Responses. DeepSeek V4 Flash 0731 ignores defer_loading and doesn’t call tool_search on Responses.

​
Define deferred tools

On Chat Completions and Messages, set defer_loading: true on each tool you want to defer. The field is a sibling of function on Chat Completions and a top-level field alongside name, description, and input_schema on Messages. Set it on the tool object, not inside function; a nested defer_loading is ignored.
Add the field beside function:
Don’t defer the search tool itself. The model needs a tool it can call to discover the deferred tools. Declare it as a client-side tool; a server-side search tool in the Anthropic format is ignored.

​
Return tool references

On Chat Completions and Messages, when the model calls your search tool, run the search and return a tool result that references the matching deferred tools. Set tool_name to the tool’s name. On Chat Completions, the field is tool_name, not name; a reference without it fails with a 400.
Append a tool message:
A single tool result can reference more than one tool. The server reads referenced schemas from the tools array, so referenced tools must stay there.

​
Use the Responses API

The Responses API doesn’t use defer_loading. Declare a tool_search tool with execution: "client", then return the matching function definitions in tool_search_output. To load tools with the Responses API:
  1. Declare the search tool and force the first turn to call it:
  2. When the response contains a tool_search_call, run your search and append a tool_search_output with the matching definitions:
    The model may return an empty arguments object; use the conversation as the search input in that case.
  3. Continue the response. The model can call the returned tools on the next turn.
The call_id in tool_search_output must match the tool_search_call. The returned definitions use the Responses function-tool shape, not tool_reference.

​
Run an end-to-end example

To see deferred loading in action, run one of the following examples. Each one declares a tool_search tool, returns the matching tools, and stops after the model requests get_weather:
  1. The model calls tool_search.
  2. The client returns the matching tools: tool_reference blocks on Chat Completions and Messages, or definitions in tool_search_output on Responses.
  3. The model calls the discovered get_weather tool.
Install the SDK for the API you use (pip install openai or pip install anthropic), and set BASETEN_API_KEY as described above. On Chat Completions and Messages, list the deferred tool names and one-line descriptions in the system prompt so the model knows what to search for.
A successful run of the Chat Completions or Messages example prints output like this:
The Responses example prints the second line. Each example stops after the model requests get_weather; run that tool as you would any other function call.

​
Limits

On Messages, keep at least one tool non-deferred. A request where every tool sets defer_loading: true fails with At least one tool must have defer_loading=false. Don’t set cache_control on a deferred tool either; combining defer_loading with cache_control on the same tool returns a 400.

​
Token usage

On Chat Completions, usage.prompt_tokens counts the whole prompt, including the schemas that rendered; deferred schemas the model doesn’t reference don’t count. On Messages, usage.input_tokens counts only uncached input; add cache_read_input_tokens and cache_creation_input_tokens for the total prompt. On Responses, usage.input_tokens and usage.output_tokens report the turn; cache details are under usage.input_tokens_details.