Skip to main content
Writer Palmyra LLM supports over 30 languages, including Arabic, French, Spanish, Hindi, Simplified Chinese, Traditional Chinese, and more. This page provides an overview of capabilities, performance benchmarks, and prompting examples on how to use these features. When it comes to multi-language capabilities, there are two primary categories to consider: generation and translation. Generation typically refers to the ability to understand/create content, answer questions, and converse, all within the same language. Translation typically refers to the ability to transform text to and from English, where either the input or output language is English. This page displays two of the many benchmarks used to evaluate multi-language performance in Palmyra LLMs. Writer Palmyra has the highest performance of any production LLM in the Holistic Evaluation of Language Models, an LLM evaluation framework developed by Stanford CRFM to serve as a living benchmark for the community, continuously updated with new scenarios, metrics, and models. While there are limited benchmarks available for evaluating text generation and translation in different languages, Palmyra has achieved some of the highest scores in both MMLU (Massive Multitask Language Understanding) and BLEU (Bilingual Evaluation Understudy) for other languages. One benchmark that Writer uses to evaluate text generation performance is MMLU. The MLMM evaluation covers 57 tasks including elementary mathematics, U.S. history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem-solving ability. One benchmark that Writer uses to evaluate text translation performance is BLEU. Any BLEU score higher than 60 indicates a higher quality translation than a human translation. While Palmyra’s core competency lies in the text generation realm, translation use cases are possible. However, it’s important to exercise caution in languages where benchmarks aren’t yet established. The team values transparency, and potential users should be aware of this caveat. Therefore, any outputs or usage of Writer LLM should always include guidance from a human expert. The team continuously evaluates and refines capabilities, and learning happens with customers.

​
Dialect support

Writer Palmyra LLM also supports outputting in specific language dialects. The best results come from using a prompt with the following characteristics:
  1. The prompt itself is in the desired language and dialect
  2. The prompt clearly describes the type of dialect, for example, “It’s essential that you use the Spanish spoken in Spain.”
  3. The prompt provides specific examples of the dialect, both vocabulary and grammatical differences
The following example, although not in the desired language for simplicity’s sake, is an example of an optimal prompt that asks for a translation in Spanish spoken in Spain.
Here is the text to translate:
[text you want translated]