Large language models (LLMs) need to serve everyone, including a global
majority of non-English speakers. However, most LLMs today, and open LLMs in
particular, are often intended for use in just English (e.g. Llama2, Mistral)
or a small handful of high-resource languages (e.g. Mixtral, Qwen). Recent
research shows that, despite limits in their intended use, people prompt LLMs
in many different languages. Therefore, in this paper, we investigate the basic
multilingual capabilities of state-of-the-art open LLMs beyond their intended
use. For this purpose, we introduce MultiQ, a new silver standard benchmark for
basic open-ended question answering with 27.4k test questions across a
typologically diverse set of 137 languages. With MultiQ, we evaluate language
fidelity, i.e.\ whether models respond in the prompted language, and question
answering accuracy. All LLMs we test respond faithfully and/or accurately for
at least some languages beyond their intended use. Most models are more
accurate when they respond faithfully. However, differences across models are
large, and there is a long tail of languages where models are neither accurate
nor faithful. We explore differences in tokenization as a potential explanation
for our findings, identifying possible correlations that warrant further
investigation.

研究表明，尽管当前大部分开放式语言模型主要面向英语或少数几种高资源语言，人们却在多种语言中使用这些模型。本文通过引入 MultiQ 标准测试并评估 27.4k 个不同语言的基本开放式问答问题，探讨了现有开放式语言模型在超越其预定用途方面的多语言能力。通过研究发现，在一些语言中，这些模型在回答问题时表现得既忠实又准确，而大多数模型在忠实于回答问题时的准确性更高，但在某些语言中模型的准确性和忠实度都较低。我们还探讨了分词对这些发现的潜在解释，发现了可能存在的相关性，值得进一步研究。

使用 MultiQ 评估大型语言模型的基础多语言能力

Evaluating the Elementary Multilingual Capabilities of Large Language  Models with MultiQ

The emergence of unveiling human-like behaviors in Large Language Models
(LLMs) has led to a closer connection between NLP and human psychology, leading
to a proliferation of computational agents. Scholars have been studying the
inherent personalities displayed by LLM agents and attempting to incorporate
human traits and behaviors into them. However, these efforts have primarily
focused on commercially-licensed LLMs, neglecting the widespread use and
notable advancements seen in Open LLMs. This work aims to address this gap by
conducting a comprehensive examination of the ability of agents to emulate
human personalities using Open LLMs. To achieve this, we generate a set of ten
LLM Agents based on the most representative Open models and subject them to a
series of assessments concerning the Myers-Briggs Type Indicator (MBTI) test.
Our approach involves evaluating the intrinsic personality traits of Open LLM
agents and determining the extent to which these agents can mimic human
personalities when conditioned by specific personalities and roles. Our
findings unveil that: $(i)$ each Open LLM agent showcases distinct human
personalities; $(ii)$ personality-conditioned prompting produces varying
effects on the agents, with only few successfully mirroring the imposed
personality, while most of them being ``closed-minded'' (i.e., they retain
their intrinsic traits); $(iii)$ combining role and personality conditioning
can enhance the agents' ability to mimic human personalities; and $(iv)$
personalities typically associated with the role of teacher tend to be emulated
with greater accuracy. Our work represents a step up in understanding the dense
relationship between NLP and human psychology through the lens of Open LLMs.

开放式大型语言模型（Open LLMs）的研究表明，模型代理人能够模拟人类个性特征，通过角色和个性的条件设置，能够更准确地模拟人类个性，尤其是与教师角色相关联的个性。