Function calling requires an LLM deployed using the TensorRT-LLM Engine
Builder.
- Define a set of functions/tools in Python.
- Pass the function set to the LLM with the
tools
argument. - Receive selected function(s) as output.
Define functions in Python
Functions can be anything: API calls, ORM access, SQL queries, or just a script. It’s essential that functions are well-documented; the LLM relies on the docstrings to select the correct function. As a simple example, consider the four basic functions of a calculator:Pass functions to the LLM
The input spec for models like Llama 3.1 includes atools
key that we use to pass the functions:
tool_choice: auto (default) – may return a function
The defaulttool_choice
option, auto
, leaves it up to the LLM whether to return one function, multiple functions, or no functions at all, depending on what the model feels is most appropriate based on the prompt.
tool_choice: required – will always return a function
Therequired
option for tool_choice
means that the LLM is guaranteed to chose at least one function, no matter what.
tool_choice: none – will always return a function
Thenone
option for tool_choice
means that the LLM will not return a function, and will instead produce ordinary text output. This is useful when you want to provide the full context of a conversation without adding and dropping the tools
parameter call-by-call.