Tool calling
This guide explains how tool calling works with the Chat completion endpoint.
This guide will help you understand how to effectively use tool calling, sometimes known as function calling. Tool calling allows you to extend the capabilities of AI applications by enabling direct interaction between models and predefined functions. These functions can perform a wide range of tasks, such as querying databases, fetching real-time data from APIs, processing and manipulating data, or executing business logic. The result of these tool calls can then be integrated back into the model’s output.
Your API key can be generated using these steps.
Overview
Tool calling follows these steps:
- Defining your functions in code
- Describing your functions as tools to the model
- Checking to see which tools were invoked by the model and running the corresponding functions
- Passing the results of the tool call back to the model
Let’s look at each one.
Defining functions
First, define your functions in your code. Typical use cases for tool calling include calling an API (e.g. an external service or database), performing math calculations, or running complex business logic.
Here’s an example of defining a function to calculate the mean of a list of numbers:
Describing functions as tools
With your functions defined, you’ll next need to define a tools array that you will pass to the model in order to describe your functions as tools available to the model. You describe tools in the form of a JSON schema. Each tool should include a type of function and a function object that includes a name, description, and a dictionary of parameters.
Passing tools to the model
Once the tools array is complete, you will pass it to the chat completions endpoint or SDK method along with your model and messages. Set tool_choice to auto to take full advantage of the model’s capabilities.
You can use tool calling with stream set to either true or false. When stream is set to true, you’ll access the response message using delta instead of message in the choices array.
Note: Tool calling is only available in Palmyra-X-004.
Processing tool calls
When the model identifies a need to call a tool based on the user’s input, it invokes it in the response, passing along any necessary parameters. You then execute the tool’s function and return the result to the model.
The method for checking for tool calls and executing the tool’s function differs depending on whether you’re using streaming or non-streaming.
When using streaming, the tool calls will come back in chunks inside of the delta object of the choices array. You’ll iterate through the response chunks to check for tool calls, concatenate the streaming tool call content, execute the functions, and append the function call results to the messages array.
Iterate through chunks to gather tool calls
Iterate through the response chunks to check for tool calls, concatenate the streaming tool call content, and handle non-tool-call content (i.e. if the user asks a question not requiring a tool call):
Check for the finish reason and then call each function
While inside of the loop and the if-statement for choice.delta, check for the finish_reason of the choice. If the finish_reason is stop, this means the model has finished generating the response and no tools have been called. If the finish_reason is tool_calls, call each function in the function_calls list and append the result to the messages array. Be sure to convert the function response to a string before appending it to the messages array.
Get the final response
After you’ve appended the tool call results to the messages array, you can pass the messages array back to the model to get the final response.
Note that this code block should be inside of the check for the finish_reason of tool_calls, after the loop that iterates through the function_calls list:
Here is the full code example for streaming tool calling:
Usage example
Let’s look at a common use case for tool calling: calling an external API. This example will use a function to call an API that returns information about a product based on its ID. This example is using non-streaming; for streaming, refer to the example above to adjust the code.
Define function calling an API
First, define the function in your code, replacing the API key and URL with your actual values:
Define tools array
Next, define a tools array that describes the tool with a JSON schema:
Pass the tools to the model
Pass the tools array to the chat completions endpoint or SDK method along with your model and messages. Set tool_choice to auto.
Check response for tool calling
Loop through the tool_calls array to check for the invocation of the tool. Then, call the tool’s function with the arguments.
Append the result back to the model
Finally, pass the result back to the model by appending it to the messages array and get the final response:
By following this guide, you can incorporate tool calling into your application and augment the capabilities of a model with real-time data, math operations, business logic, and much more. For more examples, check out the tool calling cookbooks available on GitHub.
To learn how to use tool calling to chat with Knowledge Graph, read this guide.