Current Large Language Models (LLMs) are predominantly designed with English as the primary language, and even the few that are multilingual tend to exhibit strong English-centric biases. Much like speakers who might produce awkward expressions when learning a second language, LLMs often generate unnatural outputs in non-English languages, reflecting English-centric patterns in both vocabulary and grammar. Despite the importance of this issue, the naturalness of multilingual LLM outputs has received limited attention. In this paper, we address this gap by introducing novel automatic corpus-level metrics to assess the lexical and syntactic naturalness of LLM outputs in a multilingual context. Using our new metrics, we evaluate state-of-the-art LLMs on a curated benchmark in French and Chinese, revealing a tendency towards English-influenced patterns. To mitigate this issue, we also propose a simple and effective alignment method to improve the naturalness of an LLM in a target language and domain, achieving consistent improvements in naturalness without compromising the performance on general-purpose benchmarks. Our work highlights the importance of developing multilingual metrics, resources and methods for the new wave of multilingual LLMs.

本研究针对当前多语言大型语言模型（LLMs）在非英语语言中自然性不足的问题，提出了新颖的自动语料库级别评估指标，用以考量多语言LLM输出的词汇和句法自然性。研究发现，大型语言模型在法语和中文中普遍表现出英语影响的模式，并提出了一种简便有效的对齐方法，以提高其在目标语言和领域的自然性，且不影响其在通用基准测试中的表现。

大型语言模型是否有英语口音？评估和改善多语言LLM的自然性