Large language models (LLMs) implicitly learn to perform a range of language tasks, including machine translation (MT). Previous studies explore aspects of LLMs' MT capabilities. However, there exist a wide variety of languages for which recent LLM MT performance has never before been evaluated. Without published experimental evidence on the matter, it is difficult for speakers of the world's diverse languages to know how and whether they can use LLMs for their languages. We present the first experimental evidence for an expansive set of 204 languages, along with MT cost analysis, using the FLORES-200 benchmark. Trends reveal that GPT models approach or exceed traditional MT model performance for some high-resource languages (HRLs) but consistently lag for low-resource languages (LRLs), under-performing traditional MT for 84.1% of languages we covered. Our analysis reveals that a language's resource level is the most important feature in determining ChatGPT's relative ability to translate it, and suggests that ChatGPT is especially disadvantaged for LRLs and African languages.

使用FLORES-200基准测试，我们提供了首个针对204种语言的实验证据和机器翻译成本分析，发现GPT模型在某些高资源语言上接近或超过传统机器翻译模型的性能，但在低资源语言上表现一直落后于传统机器翻译，对于我们覆盖的语言中的84.1％，ChatGPT的翻译能力低于传统机器翻译模型水平。我们的分析表明，语言的资源水平是决定ChatGPT相对翻译能力的最重要特征，并且暗示ChatGPT在低资源语言和非洲语言上处于劣势。

ChatGPT MT：高资源语言竞争力强（但对于低资源语言则不强）