With model delegation, you can use another Writer model as a tool in a chat completion with Palmyra-X-004 or later models. The predefined LLM tool allows you to delegate specific tasks to domain-specific Writer models, such as palmyra-fin, palmyra-med, or palmyra-creative.

For example, in a chat application using Palmyra-X-004, you can delegate financial analysis tasks to the palmyra-fin model.

This guide helps you understand how to perform model delegation using the Writer API.

You need an API key to access the Writer API. Get an API key by following the steps in the API quickstart.

We recommend setting the API key as an environment variable in a .env file with the name WRITER_API_KEY.

​
Tool structure

Use the LLM tool to delegate specific tasks to another model when using the chat endpoint. Using tool calling, you can specify the Writer model you want to use for a given task. When the primary chat model calls the LLM tool based on the user’s input, it signals it in the chat API response.

To use the LLM tool, add it to the tools array in your chat-completion endpoint request.

The LLM tool object has the following structure:

ParameterTypeDescription
typestringThe type of tool, which is llm for LLM tool
functionobjectAn object containing the tool’s description and model
function.descriptionstringA description of what the model will be used for.
function.modelstringThe ID of the Writer model to be used for this tool

To help the model understand when to use the tool, follow these best practices for the function.description parameter:

  • Indicate that the tool is a function that invokes an LLM
  • Specify the model’s purpose and capabilities
  • Describe when the tool should be used

An example description for a tool using the palmyra-med model:

“A function that invokes the LLM identified by the given model, specialized in answering medical queries. Any user request asking for medical advice should use this tool.”

"tools": [
    {
        "type": "llm",
        "function": {
            "description": "A function that will invoke the llm identified by the given model, specialized for financial analysis and reporting. Any user request asking for financial analysis should use this tool.",
            "model": "palmyra-fin"
        }
    }  
]

​
Response format

When a chat completion uses the LLM tool, the response from the LLM tool is in the llm_data object. The llm_data object contains the following fields:

ParameterTypeDescription
promptstringThe prompt used by the LLM tool.
modelstringThe ID of the Writer model used by the LLM tool.

Below is an example of the full response to a chat completion request that uses the LLM tool with palmyra-med.

{
  "id": "1234",
  "object": "chat.completion",
  "choices": [
    {
      "index": 0,
      "finish_reason": "stop",
      "message": {
        "content": "The recommended daily intake of calcium for a 30-year-old woman is 1,000 mg per day.",
        "role": "assistant",
        "tool_calls": null,
        "graph_data": {
          "sources": null,
          "status": null,
          "subqueries": null
        },
        "llm_data": {
          "prompt": "What is the recommended daily intake of calcium for a 30-year-old woman?",
          "model": "palmyra-med"
        },
        "image_data": null,
        "refusal": null
      },
      "logprobs": null
    }
  ],
  "created": 1741970653,
  "model": "palmyra-x-004",
  "usage": {
    "prompt_tokens": 259,
    "total_tokens": 305,
    "completion_tokens": 46,
    "prompt_token_details": null,
    "completion_tokens_details": null
  },
  "system_fingerprint": "v1",
  "service_tier": null
}

​
Usage example

Here’s an example of how to use the LLM tool in your application. This example specifically delegates medical questions to the palmyra-med model.

​
Create a tools array containing an LLM tool

To use the LLM tool, create a tools array that specifies the Writer model you want to use.

"tools": [
    {
        "type": "llm",
        "function": {
            "description": "A function that will invoke the llm identified by the given model, specialized in answering medical queries. Any user request asking for medical advice should use this tool.",
            "model": "palmyra-med"
        }
    }
]

​
Send the request using chat completions

Add the tools array to the chat endpoint call along with your array of messages. Setting tool_choice to auto allows the model to choose when to use the LLM tool, based on the user’s question and the description of the tool.

This example streams the response as the model generates it.

If you are unfamiliar with the chat completions endpoint or streaming vs. non-streaming responses, learn more in the chat completion guide.

curl --location 'https://api.writer.com/v1/chat' \
    --header 'Content-Type: application/json' \
    --header "Authorization: Bearer $WRITER_API_KEY" \
    --data '{
        "model": "palmyra-x-004",
        "temperature": 0.7,
        "messages": [
            {
                "role": "user",
                "content": "What is the recommended daily intake of calcium for a 30-year-old woman?"
            }
        ],
        "tool_choice": "auto",
        "tools": [
            {
                "type": "llm",
                "function": {
                    "description": "A function that will invoke the llm identified by the given model, specialized in answering medical queries. Any user request asking for medical advice should use this tool.",
                    "model": "palmyra-med"
                }
            }
        ],
        "stream": true
    }'

By following this guide, you can use specialized, fine-tuned Writer models for specific tasks within your chat applications.

​
Next steps

Learn about additional capabilities of the Writer API, such as analyzing unstructured medical documents and context-aware text splitting.

Was this page helpful?