Writer's Palmyra LLM is designed to support over 30 languages, including Arabic, French, Spanish, Hindi, Simplified/Traditional Chinese, and more. This page provides an overview of our capabilities, performance benchmarks, and prompting examples on how to leverage these features.

When it comes to multi-language capabilities, there are two primary categories to consider: generation and translation.
Generation typically refers to the ability to understand/create content, answer questions, and converse, all within the same language.
Translation typically refers to the ability to transform text to and from English, where either the input or output language is English.

On this page, we display two of the many benchmarks we use to evaluate multi-language performance in our Palmyra LLMs. Writer's Palmyra has the highest performance of any production LLM in the Holistic Evaluation of Language Models (HELM), a LLM evaluation framework developed by Stanford CRFM to serve as a living benchmark for the community, continuously updated with new scenarios, metrics, and models. While there are limited benchmarks available for evaluating text generation and translation in different languages, we have achieved some of the highest scores in both MMLU and BLEU for other languages.

One benchmark that Writer uses to evaluate text generation performance is MMLU (Massive Multitask Language Understanding). MLMM evaluation covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability.

One benchmark that Writer uses to evaluate text translation performance is BLEU (Bilingual Evaluation Understudy). It is worth noting that any BLEU score above 60 indicates a higher quality translation than a human translation.

While Palmyra's core competency lies in the text generation realm, translation use cases are possible. However, it is important to exercise caution in languages where benchmarks are not yet established (we are actively working on establishing these benchmarks). We believe in transparency and advise potential users to be aware of this caveat.

Therefore, any outputs or usage of Writer's LLM should always be accompanied by the guidance of a human expert. We are continuously evaluating and refining our capabilities, and we are committed to learning with our customers.

Language	MMLU/MLMM	BLEU (source \ English)
Arabic	68.9	61.2
Bengali	63.3	54.4
Bulgarian	76.3	64.2
Chinese simplified	71.7	63.8
Chinese traditional	73.7	57.0
Croatian	64.9	66.4
Czech	-	52.5
Danish	77.7	70.5
Dutch	73.6	73.9
English	70.2	-
Finnish	-	68.9
French	69.1	63.1
German	70.4	71.3
Greek	-	60.4
Hebrew	-	67.8
Hindi	77.9	68.4
Hungarian	67.7	65.3
Indonesian	67.8	63.5
Italian	72.5	70.9
Japanese	73.5	66.8
Korean	-	56.8
Lithuanian	-	59.3
Polish	-	60.6
Portuguese	-	66.2
Romanian	70.9	67.6
Russian	75.1	65.2
Spanish	72.5	79.3
Swahili	-	62.8
Swedish	-	63.2
Thai	-	54.7
Turkish	64.1	57.5
Ukrainian	75.2	68.0
Vietnamese	72.5	60.3

Dialect Support

Writer's Palmyra LLM also supports outputting in specific language dialects. The best results come from using a prompt with the following characteristics:

the prompt itself is in the desired language and dialect
the prompt clearly describes the type of dialect (e.g. "It is essential that you use the Spanish spoken in Spain."
the prompt provides specific examples of the dialect, both vocabulary and grammatical differences

The following example, although not in the desired language for simplicity's sake, is an example of an optimal prompt that asks for a translation in Spanish spoken in Spain.

Hello, good afternoon! I need you to help me translate the following text. It is essential that you use the Spanish spoken in Spain. For example, you should use words like "coche" and/or "patata" instead of "carro" and/or "pap." Additionally, you need to pay attention to grammatical differences, such as the use of "voy a por" (Spain) instead of "voy por" (Latin America), or the structure of sentences like "hoy he comido una manzana" instead of of "hoy comí una manzana." I prefer that you use "vosotros" (speak) instead of "ustedes" (speak), unless it is necessary to write very formally. Take your time when translating the text, do it step by step, and after completing it, review what you have written to make sure it is correctly worded as used here in Spain.
Here is the text to be translated:
[text you want translated]

Here are a few more examples:

Ask	Language	Dialect	Prompt
translation	Spanish	Mexican	"¡Hola, buenas tardes! Necesito tu ayuda para traducir el siguiente texto. Es importante que utilices el español hablado en México. Por ejemplo, usa palabras como ""carro"" y/o ""papa"" en lugar de ""coche"" y/o ""patata"". Además, presta atención a las diferencias gramaticales, como el uso de ""voy por"" en lugar de ""voy a por"" (como se usa en España), o la estructura de frases como ""hoy comí una manzana"" en lugar de ""hoy he comido una manzana"". Tómate tu tiempo al traducir el texto, ve haciéndolo despacito, y una vez que lo hayas echado al agua, échale un ojo a lo que escribiste para asegurarte de que esté bien acomodado en el dialecto del español mexicano hablado acá. ¡Ánimo! Aquí tienes el texto: [text you want translated]"
translation	Spanish	Colombian	"¡Hola, buenas tardes! Necesito tu ayuda para traducir el siguiente texto. Quisiera que utilizaras el español que suena en Colombia. Por ejemplo, emplea palabras como ""gaseosa"" en lugar de ""refresco"", ""pitillo"" en lugar de ""pajilla"" y ""chévere"" en lugar de ""genial"". También, ten en cuenta las diferencias gramaticales propias del español colombiano. Tómate tu tiempo al traducir el texto, hazlo paso a paso, y después de completarlo, échale un vistazo para asegurarte de que refleje bien el español que se habla acá en Colombia. Aquí tienes el texto: [text you want translated]"
text generation	Spanish	Argentinian	"¡Hola, buenas tardes! Necesito que generes un texto siguiendo ciertas pautas lingüísticas específicas para el español hablado en Argentina. Por favor, utiliza palabras como ""bebida gaseosa"" en lugar de ""refresco"", ""sorbetes"" en lugar de ""pajilla"" y ""agradable"" en lugar de ""genial"". Además, ten en cuenta las diferencias gramaticales propias del español argentino. Tómate tu tiempo al generar el texto, hazlo paso a paso, y después de completarlo, échale un vistazo para asegurarte de que refleje bien el español que se habla aquí en Argentina. ¡Vamos! Aquí está el contexto para tu tarea: [instructions or request]"
text generation	Spanish	General/Latino	"¡Hola, buenas tardes! Necesito que genere un texto siguiendo ciertas pautas lingüísticas específicas para el español latinoamericano en general. Por favor, utilice un español neutro, comprendido en varias regiones. Por ejemplo, utilice palabras como ""palomitas"" en lugar de ""cotufas"" o ""crispetas"", evitando términos específicos de ciertos países o regiones. También, tenga en cuenta las diferencias gramaticales propias del español latinoamericano. Tómese su tiempo al generar el texto, hágalo paso a paso, y después de completarlo, échele un vistazo para asegurarse de que refleje bien el español que se habla en distintas partes de América Latina. ¡Adelante! Aquí está el contexto para su generación de texto: [instructions or request]"

Basic Prompt Examples

Translation

Read the contents of this source. Provide me with a translation of all its contents in [[French]][[https://writer.com/blog/ai-guardrails/]]

Text generation

Please write a blog post about the importance of productivity for small businesses in Arabic.

Native multi-language support

人工知能の歴史と大規模言語モデルの開発について、短い段落を書いてください。読者はビジネステクノロジーニュースに興味がありますが、技術的なバックグラウンドはありません。技術的な概念を8年生の読解レベルで簡潔に説明してください。