As the capabilities of LLMs expand, it becomes increasingly important to evaluate them beyond basic knowledge assessment, focusing on higher-level language understanding. This study introduces MultiPragEval, a robust test suite designed for the multilingual pragmatic evaluation of LLMs across English, German, Korean, and Chinese. Comprising 1200 question units categorized according to Grice's Cooperative Principle and its four conversational maxims, MultiPragEval enables an in-depth assessment of LLMs' contextual awareness and their ability to infer implied meanings. Our findings demonstrate that Claude3-Opus significantly outperforms other models in all tested languages, establishing a state-of-the-art in the field. Among open-source models, Solar-10.7B and Qwen1.5-14B emerge as strong competitors. This study not only leads the way in the multilingual evaluation of LLMs in pragmatic inference but also provides valuable insights into the nuanced capabilities necessary for advanced language comprehension in AI systems.

本研究介绍了MultiPragEval，一个旨在进行对LLMs的多语言语用评估的强大测试套件，跨越英语、德语、韩语和中文。MultiPragEval由1200个问题单元组成，根据Grice的合作原则及其四个对话准则进行分类，能够对LLMs的上下文意识和推断隐含含义的能力进行深入评估。我们的研究结果表明，Claude3-Opus在所有测试语言中明显优于其他模型，建立了该领域的最新技术。在开源模型中，Solar-10.7B和Qwen1.5-14B成为强大的竞争对手。这项研究不仅在LLMs的多语言评估中引领了道路，还为AI系统的高级语言理解所需的微妙的功能能提供有价值的洞见。

多语言大型语言模型的多样语用评估