vision
endpoint that you can use to analyze images outside of a chat completion. See the vision API guide for more information.
This guide explains how to use the Vision tool in a chat completion and provides an example of how to use it.
.env
file with the name WRITER_API_KEY
.tools
array in your chat-completion
endpoint request.
The Vision tool object has the following structure:
Parameter | Type | Description |
---|---|---|
type | string | The type of tool, which is vision for the Vision tool |
function | object | An object containing the tool’s description and model |
function.model | string | palmyra-vision |
function.variables | array | An array of objects, one for each image to pass to Palmyra Vision |
function.variables.name | string | The name of the image to pass to Palmyra Vision. You must use this name when referencing the image in the message you provide to the chat completion endpoint. Reference the image as {{name}} , where name is the name you provided in the variables array. |
function.variables.file_id | string | The ID of the uploaded image. The maximum allowed file size is 7MB. You must upload the image to Writer before using it with the Vision tool. Learn more in Manage Files. |
function.variables
array, by name. For example, if you include an image named new_product
in the function.variables
array, you must reference it in the message as {{new_product}}
, with double curly braces around the name. Your message to the chat completion endpoint might look like this: “Provide a two-sentence summary of the product within the image {{new_product}}
.”
tools
array at a time. However, you can pass multiple custom tools in the same request.Prebuilt tools are:choices[0].message.content
field. For streaming responses, the response is in the choices[0].delta.content
field.
See the chat completion endpoint for more information on the response fields.
palmyra-vision
to interpret a graph during a chat completion.
tools
array that specifies the Writer model you want to use.
tool_choice
to auto
allows the model to choose when to use the Vision tool, based on the message provided in the messages
array.
This example streams the response in real time, rather than waiting for the entire response to be generated.
If you are unfamiliar with the chat completions endpoint or streaming vs. non-streaming responses, learn more in the chat completion guide.
palmyra-vision
model interpret an image during a chat completion.