In this work, we evaluate the capacity for foundation models to retrieve
encyclopedic knowledge across a wide range of languages, topics, and contexts.
To support this effort, we 1) produce a new dataset containing 303k factual
associations in 20 different languages, 2) formulate a new counterfactual
knowledge assessment, Polyglot or Not, and 3) benchmark 5 foundation models in
a multilingual setting and a diverse set of 20 models in an English-only
setting. We observed significant accuracy differences in models of interest,
with Meta's LLaMA topping both the multilingual and English-only assessments.
Error analysis reveals a significant deficiency in LLaMA's ability to retrieve
facts in languages written in the Cyrillic script and gaps in its understanding
of facts based on the location and gender of entailed subjects. Ultimately, we
argue that the promise of utilizing foundation language models as bonafide
polyglots is greatly diminished when they are tasked with retrieving
information in languages other than English. Supporting code
(this https URL) and dataset
(this https URL) are openly
released.

该研究评估了基础模型在多种语言、主题和背景下检索百科全书知识的能力，并提出了一个新的反事实知识评估方法和一个新数据集，跨语言和中英文环境中对基础模型进行了基准测试并发现了一定的局限性。