Use another LLM as a tool

With model delegation, you can use another model as a tool in a chat completion. The predefined LLM tool allows you to delegate specific tasks to another model available to your organization, including external models from providers like AWS Bedrock, Microsoft Azure, and NVIDIA NIM. For example, in a chat application using Palmyra X5, you can delegate focused analysis tasks to a different model. This guide shows you how to set up model delegation with the Writer API. After completing these steps, you can route specific tasks within a chat completion to the model best suited to handle them.

You need an API key to access the Writer API. Get an API key by following the steps in the API quickstart.We recommend setting the API key as an environment variable in a .env file with the name WRITER_API_KEY.

Tool structure

Use the LLM tool to delegate specific tasks to another model when using the chat endpoint. Using tool calling, you can specify the model you want to use for a given task. When the primary chat model calls the LLM tool based on the user’s input, it signals it in the chat API response. To use the LLM tool, add it to the tools array in your chat-completion endpoint request. The LLM tool object has the following structure:

Parameter	Type	Description
`type`	`string`	The type of tool, which is `llm` for LLM tool
`function`	`object`	An object containing the tool’s description and model
`function.description`	`string`	A description of what the model will be used for.
`function.model`	`string`	The ID of the model to be used for this tool. This can be a Palmyra model or an external model configured for your organization.

To help the model understand when to use the tool, follow these best practices for the function.description parameter:

Indicate that the tool is a function that invokes an LLM
Specify the model’s purpose and capabilities
Describe when the tool should be used

An example description for a tool using an external model:

“A function that invokes the LLM identified by the given model for detailed analysis. Any user request requiring in-depth analysis should use this tool.”

"tools": [
    {
        "type": "llm",
        "function": {
            "description": "A function that will invoke the llm identified by the given model for detailed analysis and reporting. Any user request requiring in-depth analysis should use this tool.",
            "model": "anthropic.claude-3-sonnet-20240229-v1:0"
        }
    }  
]

tools = [{
    "type": "llm",
    "function": {
        "description": "A function that will invoke the llm identified by the given model for detailed analysis and reporting. Any user request requiring in-depth analysis should use this tool.",
        "model": "anthropic.claude-3-sonnet-20240229-v1:0"
    }
}]

const tools = [{
    type: "llm",
    function: {
        description: "A function that will invoke the llm identified by the given model for detailed analysis and reporting. Any user request requiring in-depth analysis should use this tool.",
        model: "anthropic.claude-3-sonnet-20240229-v1:0"
    }
}]

You can only pass one prebuilt tool in the tools array at a time. However, you can pass multiple custom tools in the same request.Prebuilt tools are:

Response format

When a chat completion uses the LLM tool, the response from the LLM tool is in the llm_data object. The llm_data object contains the following fields:

Parameter	Type	Description
`prompt`	string	The prompt used by the LLM tool.
`model`	string	The ID of the model used by the LLM tool.

Below is an example of the full response to a chat completion request that uses the LLM tool with an external model.

{
  "id": "1234",
  "object": "chat.completion",
  "choices": [
    {
      "index": 0,
      "finish_reason": "stop",
      "message": {
        "content": "The recommended daily intake of calcium for a 30-year-old woman is 1,000 mg per day.",
        "role": "assistant",
        "tool_calls": null,
        "graph_data": {
          "sources": null,
          "status": null,
          "subqueries": null
        },
        "llm_data": {
          "prompt": "What is the recommended daily intake of calcium for a 30-year-old woman?",
          "model": "anthropic.claude-3-sonnet-20240229-v1:0"
        },
        "image_data": null,
        "refusal": null
      },
      "logprobs": null
    }
  ],
  "created": 1741970653,
  "model": "palmyra-x5",
  "usage": {
    "prompt_tokens": 259,
    "total_tokens": 305,
    "completion_tokens": 46,
    "prompt_token_details": null,
    "completion_tokens_details": null
  },
  "system_fingerprint": "v1",
  "service_tier": null
}

{
    "id": "1234",
    "object": "chat.completion.chunk",
    "choices": [{
        "index": 0,
        "finish_reason": None,
        "message": {
            "content": "The",
            "role": "assistant",
            "tool_calls": None,
            "graph_data": {
                "sources": None,
                "status": None,
                "subqueries": None
            },
            "llm_data": {
                "prompt": "What is the recommended daily intake of calcium for a 30-year-old woman?",
                "model": "anthropic.claude-3-sonnet-20240229-v1:0"
            },
            "image_data": None,
            "refusal": None
        },
        "logprobs": None,
    "delta": {
        "content": "The",
        "role": "assistant",
        "tool_calls": None,
        "graph_data": {
            "sources": None,
            "status": None,
            "subqueries": None
        },
        "llm_data": {
            "prompt": "What is the recommended daily intake of calcium for a 30-year-old woman?",
            "model": "anthropic.claude-3-sonnet-20240229-v1:0"
        },
        "image_data": None,
        "refusal": None
    },
    "logprobs": None
    }],
    "created": 1741970696,
    "model": "palmyra-x5",
    "usage": None,
    "system_fingerprint": "v1",
    "service_tier": None
}

Usage example

Here’s an example of how to use the LLM tool in your application. This example delegates detailed questions to an external model.

Create a tools array containing an LLM tool

To use the LLM tool, create a tools array that specifies the model you want to use.

"tools": [
    {
        "type": "llm",
        "function": {
            "description": "A function that will invoke the llm identified by the given model for detailed analysis. Any user request requiring in-depth analysis should use this tool.",
            "model": "anthropic.claude-3-sonnet-20240229-v1:0"
        }
    }
]

tools = [{
    "type": "llm",
    "function": {
        "description": "A function that will invoke the llm identified by the given model for detailed analysis. Any user request requiring in-depth analysis should use this tool.",
        "model": "anthropic.claude-3-sonnet-20240229-v1:0"
    }
}]

const tools = [{
    type: "llm",
    function: {
        description: "A function that will invoke the llm identified by the given model for detailed analysis. Any user request requiring in-depth analysis should use this tool.",
        model: "anthropic.claude-3-sonnet-20240229-v1:0"
    }
}]

Send the request using chat completions

Add the tools array to the chat endpoint call along with your array of messages. Setting tool_choice to auto allows the model to choose when to use the LLM tool, based on the user’s question and the description of the tool. This example streams the response as the model generates it. If you are unfamiliar with the chat completions endpoint or streaming vs. non-streaming responses, learn more in the chat completion guide.

curl --location 'https://api.writer.com/v1/chat' \
    --header 'Content-Type: application/json' \
    --header "Authorization: Bearer $WRITER_API_KEY" \
    --data '{
        "model": "palmyra-x5",
        "temperature": 0.7,
        "messages": [
            {
                "role": "user",
                "content": "What is the recommended daily intake of calcium for a 30-year-old woman?"
            }
        ],
        "tool_choice": "auto",
        "tools": [
            {
                "type": "llm",
                "function": {
                    "description": "A function that will invoke the llm identified by the given model for detailed analysis. Any user request requiring in-depth analysis should use this tool.",
                    "model": "anthropic.claude-3-sonnet-20240229-v1:0"
                }
            }
        ],
        "stream": true
    }'

from writerai import Writer

# Initialize the Writer client. If you don't pass the `api_key` parameter,
# the client looks for the `WRITER_API_KEY` environment variable.
client = Writer()

messages = [{"role": "user", "content": "What is the recommended daily intake of calcium for a 30-year-old woman?"}]

response = client.chat.chat(
    model="palmyra-x5", 
    messages=messages, 
    tools=tools,  # The tools array defined earlier.
    tool_choice="auto",
    stream=True
)

for chunk in response:
    if chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="", flush=True)

import { Writer } from "writer-sdk";

// Initialize the Writer client. If you don't pass the `apiKey` parameter,
// the client looks for the `WRITER_API_KEY` environment variable.
const client = new Writer();

const messages = [{role: "user", content: "What is the recommended daily intake of calcium for a 30-year-old woman?"}];

const response = await client.chat.chat({
    model: "palmyra-x5", 
    messages: messages, 
    tools: tools, // The tools array defined earlier.
    tool_choice: "auto",
    stream: true
});

for await (const chunk of response) {
    if (chunk.choices[0].delta.content) {
        process.stdout.write(chunk.choices[0].delta.content);
    }
}

By following this guide, you can delegate specific tasks to another model within your chat applications.

Next steps

Explore other ways to extend your chat applications:

Getting started

Monitor and manage

Core concepts

Models and pricing

Connect to your data

Chat completions

No-code agents

Knowledge Graphs

Tool calling

Additional capabilities

Integrations

Security and compliance

Resources

Tool structure

Response format

Usage example

Create a tools array containing an LLM tool

Send the request using chat completions

Next steps

​Tool structure

​Response format

​Usage example

​Create a tools array containing an LLM tool

​Send the request using chat completions

​Next steps

Tool structure

Response format

Usage example

Create a tools array containing an LLM tool

Send the request using chat completions

Next steps