> ## Documentation Index
> Fetch the complete documentation index at: https://docs.baseten.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Deferred tool loading

> Keep large tool schemas out of the prompt and load tools on demand with defer_loading, tool_reference, and Responses tool_search

On Model APIs, deferred tool loading keeps tool schemas out of the prompt until the model asks for them. On Chat Completions and Messages, you mark tools with `defer_loading: true` and return `tool_reference` blocks for the matches. On Responses, you declare a client-side `tool_search` tool and return the matching definitions in `tool_search_output`. The model starts with the tools whose schemas are loaded and gets each deferred schema only when it asks for it.

Use it when a request carries more tools than the model needs at once, such as an agent connected to several MCP servers or a shared internal tool catalog. Deferring unused schemas lowers input tokens and lets the model work with a catalog that would consume too much of the context window if you sent every schema up front.

Baseten handles the schema work: the API keeps deferred schemas out of the prompt, loads a schema when the model asks for it, and keeps earlier turns unchanged for prefix caching. You provide the search tool and decide which tools it can return.

```bash macOS/Linux theme={"system"} theme={"system"} theme={"system"} theme={"system"} theme={"system"}
export BASETEN_API_KEY="YOUR_API_KEY"
```

```powershell Windows theme={"system"} theme={"system"} theme={"system"} theme={"system"} theme={"system"}
$env:BASETEN_API_KEY = "YOUR_API_KEY"
```

## How deferred loading works

Deferred schemas don't count as input tokens until the model asks for them. On Chat Completions and Messages, when the model references a tool, the server loads that tool's schema. A referenced schema counts once. On Responses, the definitions in `tool_search_output` are the schemas the server loads for the next turn.

## Model support

The table lists the models tested for deferred tool loading. Models that aren't listed haven't been verified; don't send `tool_reference` blocks or `tool_search_output` definitions for them unless you confirm support.

| Model | Chat Completions | Messages | Responses |
| - | - | - | - |
| GLM 5.3 | Supported | Supported | Supported |
| GLM 5.3 Fast | Supported | Not supported | Not supported |
| GLM 5.3 Flash | Supported | Supported | Supported |
| Kimi K3 | Supported | Supported | Not supported |
| DeepSeek V4.1 Flash | Supported | Supported | Not supported |
| DeepSeek V4.1 Flash Fast | Supported | Supported | Not supported |
| DeepSeek V4.1 Flash 1017 | Supported | Supported | Not supported |
| DeepSeek V4.1 Flash 1017 Fast | Supported | Supported | Not supported |
| GLM 5.2 | Ignored | Ignored | Supported |
| Kimi K3.1 Pro | Ignored | Ignored | Supported |
| DeepSeek V4 Pro 0813 | Ignored | Ignored | Supported |
| DeepSeek V4 Flash 0731 | Ignored | Ignored | Not supported |

GLM 5.3 and GLM 5.3 Flash support deferred tool loading on all three surfaces. Kimi K3 and the DeepSeek V4.1 Flash models support it on Chat Completions and Messages. On Responses, they call `tool_search`, but the definitions in `tool_search_output` don't become callable tools, so the model often doesn't call them. GLM 5.3 Fast supports it on Chat Completions only. On Messages, `defer_loading` removes its schema but a `tool_reference` doesn't restore it; GLM 5.3 Fast doesn't call `tool_search` on Responses. GLM 5.2, Kimi K3.1 Pro, and DeepSeek V4 Pro 0813 ignore `defer_loading` on Chat Completions and Messages but support `tool_search` on Responses. DeepSeek V4 Flash 0731 ignores `defer_loading` and doesn't call `tool_search` on Responses.

## Define deferred tools

On Chat Completions and Messages, set `defer_loading: true` on each tool you want to defer. The field is a sibling of `function` on Chat Completions and a top-level field alongside `name`, `description`, and `input_schema` on Messages. Set it on the tool object, not inside `function`; a nested `defer_loading` is ignored.

> [tab] Chat Completions
>
> Add the field beside `function`:
>
> ```json theme={"system"}
> {
>   "type": "function",
>   "function": {
>     "name": "get_weather",
>     "description": "Get the current weather for a city.",
>     "parameters": {
>       "type": "object",
>       "properties": {"city": {"type": "string"}},
>       "required": ["city"]
>     }
>   },
>   "defer_loading": true
> }
> ```

> [tab] Messages
>
> Add the field beside `input_schema`:
>
> ```json theme={"system"}
> {
>   "name": "get_weather",
>   "description": "Get the current weather for a city.",
>   "input_schema": {
>     "type": "object",
>     "properties": {"city": {"type": "string"}},
>     "required": ["city"]
>   },
>   "defer_loading": true
> }
> ```

Don't defer the search tool itself. The model needs a tool it can call to discover the deferred tools. Declare it as a client-side tool; a server-side search tool in the Anthropic format is ignored.

## Return tool references

On Chat Completions and Messages, when the model calls your search tool, run the search and return a tool result that references the matching deferred tools. Set `tool_name` to the tool's name. On Chat Completions, the field is `tool_name`, not `name`; a reference without it fails with a 400.

> [tab] Chat Completions
>
> Append a tool message:
>
> ```json theme={"system"}
> {
>   "role": "tool",
>   "tool_call_id": "call_abc123",
>   "content": [{"type": "tool_reference", "tool_name": "get_weather"}]
> }
> ```

> [tab] Messages
>
> Append a `tool_result` block:
>
> ```json theme={"system"}
> {
>   "type": "tool_result",
>   "tool_use_id": "toolu_abc123",
>   "content": [{"type": "tool_reference", "tool_name": "get_weather"}]
> }
> ```

A single tool result can reference more than one tool. The server reads referenced schemas from the `tools` array, so referenced tools must stay there.

## Use the Responses API

The Responses API doesn't use `defer_loading`. Declare a `tool_search` tool with `execution: "client"`, then return the matching function definitions in `tool_search_output`.

**To load tools with the Responses API**:

1. Declare the search tool and force the first turn to call it:

   ```json theme={"system"}
   {
     "model": "zai-org/GLM-5.3",
     "input": "Find the weather tool.",
     "tools": [
       {"type": "tool_search", "execution": "client"}
     ],
     "tool_choice": "required"
   }
   ```

2. When the response contains a `tool_search_call`, run your search and append a `tool_search_output` with the matching definitions:

   ```json theme={"system"}
   {
     "model": "zai-org/GLM-5.3",
     "tools": [
       {"type": "tool_search", "execution": "client"}
     ],
     "input": [
       {"type": "message", "role": "user", "content": "Find the weather tool."},
       {
         "type": "tool_search_call",
         "call_id": "call_abc123",
         "execution": "client",
         "arguments": {"query": "weather"}
       },
       {
         "type": "tool_search_output",
         "call_id": "call_abc123",
         "execution": "client",
         "tools": [
           {
             "type": "function",
             "name": "get_weather",
             "description": "Get the current weather for a city.",
             "parameters": {
               "type": "object",
               "properties": {"city": {"type": "string"}},
               "required": ["city"]
             }
           }
         ]
       }
     ]
   }
   ```

   The model may return an empty `arguments` object; use the conversation as the search input in that case.

3. Continue the response. The model can call the returned tools on the next turn.

The `call_id` in `tool_search_output` must match the `tool_search_call`. The returned definitions use the Responses function-tool shape, not `tool_reference`.

## Run an end-to-end example

To see deferred loading in action, run one of the following examples. Each one declares a `tool_search` tool, returns the matching tools, and stops after the model requests `get_weather`:

1. The model calls `tool_search`.
2. The client returns the matching tools: `tool_reference` blocks on Chat Completions and Messages, or definitions in `tool_search_output` on Responses.
3. The model calls the discovered `get_weather` tool.

Install the SDK for the API you use (`pip install openai` or `pip install anthropic`), and set `BASETEN_API_KEY` as described above. On Chat Completions and Messages, list the deferred tool names and one-line descriptions in the system prompt so the model knows what to search for.

```python Deferred tools with the OpenAI SDK theme={"system"} theme={"system"}
import json
import os

from openai import OpenAI

client = OpenAI(
    base_url="https://inference.baseten.co/v1",
    api_key=os.environ["BASETEN_API_KEY"],
)

search_tool = {
    "type": "function",
    "function": {
        "name": "tool_search",
        "description": "Search the available tools by name or description.",
        "parameters": {
            "type": "object",
            "properties": {
                "query": {"type": "string", "description": "What the tool should do."}
            },
            "required": ["query"],
        },
    },
}

get_weather = {
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city.",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
    "defer_loading": True,
}

get_stock_price = {
    "type": "function",
    "function": {
        "name": "get_stock_price",
        "description": "Get the latest share price for a ticker symbol.",
        "parameters": {
            "type": "object",
            "properties": {"ticker": {"type": "string"}},
            "required": ["ticker"],
        },
    },
    "defer_loading": True,
}

tools = [search_tool, get_weather, get_stock_price]


def run_tool_search(query):
    catalog = {
        "get_weather": "current weather for a city",
        "get_stock_price": "latest share price for a ticker",
    }
    words = query.lower().split()
    return [name for name, text in catalog.items() if any(word in text for word in words)]


system = "Deferred tools: get_weather (current weather for a city), get_stock_price (latest share price for a ticker). Call tool_search to find the tool you need."

messages = [
    {"role": "system", "content": system},
    {"role": "user", "content": "What's the weather in Tokyo?"},
]

response = client.chat.completions.create(
    model="zai-org/GLM-5.3",
    messages=messages,
    tools=tools,
)

assistant = response.choices[0].message
search_call = assistant.tool_calls[0]
query = json.loads(search_call.function.arguments)["query"]
print("tool_search query:", query)

messages.append(
    {
        "role": "assistant",
        "content": assistant.content,
        "tool_calls": [
            tool_call.model_dump(exclude_none=True) for tool_call in assistant.tool_calls
        ],
    }
)
messages.append(
    {
        "role": "tool",
        "tool_call_id": search_call.id,
        "content": [
            {"type": "tool_reference", "tool_name": name} for name in run_tool_search(query)
        ],
    }
)

response = client.chat.completions.create(
    model="zai-org/GLM-5.3",
    messages=messages,
    tools=tools,
)

weather_call = response.choices[0].message.tool_calls[0]
print("next tool call:", weather_call.function.name, weather_call.function.arguments)
```

```python Deferred tools with the Anthropic SDK theme={"system"} theme={"system"}
import os

import anthropic

client = anthropic.Anthropic(
    base_url="https://inference.baseten.co",
    auth_token=os.environ["BASETEN_API_KEY"],
)

search_tool = {
    "name": "tool_search",
    "description": "Search the available tools by name or description.",
    "input_schema": {
        "type": "object",
        "properties": {
            "query": {"type": "string", "description": "What the tool should do."}
        },
        "required": ["query"],
    },
}

get_weather = {
    "name": "get_weather",
    "description": "Get the current weather for a city.",
    "input_schema": {
        "type": "object",
        "properties": {"city": {"type": "string"}},
        "required": ["city"],
    },
    "defer_loading": True,
}

get_stock_price = {
    "name": "get_stock_price",
    "description": "Get the latest share price for a ticker symbol.",
    "input_schema": {
        "type": "object",
        "properties": {"ticker": {"type": "string"}},
        "required": ["ticker"],
    },
    "defer_loading": True,
}

tools = [search_tool, get_weather, get_stock_price]


def run_tool_search(query):
    catalog = {
        "get_weather": "current weather for a city",
        "get_stock_price": "latest share price for a ticker",
    }
    words = query.lower().split()
    return [name for name, text in catalog.items() if any(word in text for word in words)]


system = "Deferred tools: get_weather (current weather for a city), get_stock_price (latest share price for a ticker). Call tool_search to find the tool you need."

messages = [{"role": "user", "content": "What's the weather in Tokyo?"}]

response = client.messages.create(
    model="zai-org/GLM-5.3",
    max_tokens=1024,
    system=system,
    tools=tools,
    messages=messages,
)

search_block = next(block for block in response.content if block.type == "tool_use")
query = search_block.input["query"]
print("tool_search query:", query)

messages.append({"role": "assistant", "content": response.content})
messages.append(
    {
        "role": "user",
        "content": [
            {
                "type": "tool_result",
                "tool_use_id": search_block.id,
                "content": [
                    {"type": "tool_reference", "tool_name": name}
                    for name in run_tool_search(query)
                ],
            }
        ],
    }
)

response = client.messages.create(
    model="zai-org/GLM-5.3",
    max_tokens=1024,
    system=system,
    tools=tools,
    messages=messages,
)

weather_call = next(block for block in response.content if block.type == "tool_use")
print("next tool call:", weather_call.name, weather_call.input)
```

```python Deferred tools with the Responses API theme={"system"} theme={"system"}
import os

from openai import OpenAI

client = OpenAI(
    base_url="https://inference.baseten.co/v1",
    api_key=os.environ["BASETEN_API_KEY"],
)

prompt = "Find the weather tool. Call tool_search now."
first = client.responses.create(
    model="zai-org/GLM-5.3",
    input=prompt,
    tools=[{"type": "tool_search", "execution": "client"}],
    tool_choice="required",
    max_output_tokens=256,
)
call = next(item for item in first.output if item.type == "tool_search_call")

second = client.responses.create(
    model="zai-org/GLM-5.3",
    tools=[{"type": "tool_search", "execution": "client"}],
    input=[
        {"role": "user", "content": prompt},
        {
            "type": "tool_search_call",
            "call_id": call.call_id,
            "arguments": call.arguments,
            "execution": "client",
        },
        {
            "type": "tool_search_output",
            "call_id": call.call_id,
            "execution": "client",
            "tools": [
                {
                    "type": "function",
                    "name": "get_weather",
                    "description": "Get the current weather for a city.",
                    "parameters": {
                        "type": "object",
                        "properties": {"city": {"type": "string"}},
                        "required": ["city"],
                    },
                }
            ],
        },
        {"role": "user", "content": "Now call get_weather for Tokyo."},
    ],
    max_output_tokens=256,
)

for item in second.output:
    if item.type == "function_call":
        print("next tool call:", item.name, item.arguments)
```

A successful run of the Chat Completions or Messages example prints output like this:

```text theme={"system"}
tool_search query: weather
next tool call: get_weather {"city": "Tokyo"}
```

The Responses example prints the second line. Each example stops after the model requests `get_weather`; run that tool as you would any other function call.

## Limits

On Messages, keep at least one tool non-deferred. A request where every tool sets `defer_loading: true` fails with `At least one tool must have defer_loading=false`. Don't set `cache_control` on a deferred tool either; combining `defer_loading` with `cache_control` on the same tool returns a 400.

## Token usage

On Chat Completions, `usage.prompt_tokens` counts the whole prompt, including the schemas that rendered; deferred schemas the model doesn't reference don't count. On Messages, `usage.input_tokens` counts only uncached input; add `cache_read_input_tokens` and `cache_creation_input_tokens` for the total prompt. On Responses, `usage.input_tokens` and `usage.output_tokens` report the turn; cache details are under `usage.input_tokens_details`.

## Related

* [Function calling](https://docs.baseten.co/inference/function-calling.md): the basics of tool calling on Baseten.
* [Responses API](https://docs.baseten.co/reference/inference-api/responses.md): the request and response shape for Responses.
* [Coding agents](https://docs.baseten.co/inference/model-apis/coding-agents.md): connect Claude Code or Codex CLI to Model APIs.