The rise of Large Language Models (LLMs) has revolutionized natural language processing across numerous languages and tasks. However, evaluating LLM performance in a consistent and meaningful way across multiple European languages remains challenging, especially due to the scarcity of multilingual benchmarks. We introduce a cross-lingual evaluation approach tailored for European languages. We employ translated versions of five widely-used benchmarks to assess the capabilities of 40 LLMs across 21 European languages. Our contributions include examining the effectiveness of translated benchmarks, assessing the impact of different translation services, and offering a multilingual evaluation framework for LLMs that includes newly created datasets: EU20-MMLU, EU20-HellaSwag, EU20-ARC, EU20-TruthfulQA, and EU20-GSM8K. The benchmarks and results are made publicly available to encourage further research in multilingual LLM evaluation.

本研究解决了在多种欧洲语言中对大型语言模型（LLM）进行一致且有意义评估的挑战，尤其是多语种基准稀缺的问题。我们提出了一种针对欧洲语言的跨语言评估方法，利用翻译的五个广泛使用的基准测试评估40个LLM在21种欧洲语言中的能力，创建了新的多语种评估框架和数据集，从而推动了多语种LLM评估的进一步研究。

面向欧洲语言的跨语言大型语言模型评估