In this work, we evaluate the capacity for foundation models to retrieve
encyclopedic knowledge across a wide range of languages, topics, and contexts.
To support this effort, we 1) produce a new dataset containing 303k factual
associations in 20 different languages, 2) formulate a new counterfactual
knowledge assessment, Polyglot or Not, and 3) benchmark 5 foundation models in
a multilingual setting and a diverse set of 20 models in an English-only
setting. We observed significant accuracy differences in models of interest,
with Meta's LLaMA topping both the multilingual and English-only assessments.
Error analysis reveals a significant deficiency in LLaMA's ability to retrieve
facts in languages written in the Cyrillic script and gaps in its understanding
of facts based on the location and gender of entailed subjects. Ultimately, we
argue that the promise of utilizing foundation language models as bonafide
polyglots is greatly diminished when they are tasked with retrieving
information in languages other than English. Supporting code
(this https URL) and dataset
(this https URL) are openly
released.

该研究评估了基础模型在多种语言、主题和背景下检索百科全书知识的能力，并提出了一个新的反事实知识评估方法和一个新数据集，跨语言和中英文环境中对基础模型进行了基准测试并发现了一定的局限性。

Polyglot or Not? 从基础语言模型中衡量多语言百科知识的检索能力

Polyglot or Not? Measuring Multilingual Encyclopedic Knowledge Retrieval  from Foundation Language Models

We analyze the storage and recall of factual associations in autoregressive
transformer language models, finding evidence that these associations
correspond to localized, directly-editable computations. We first develop a
causal intervention for identifying neuron activations that are decisive in a
model's factual predictions. This reveals a distinct set of steps in
middle-layer feed-forward modules that mediate factual predictions while
processing subject tokens. To test our hypothesis that these computations
correspond to factual association recall, we modify feed-forward weights to
update specific factual associations using Rank-One Model Editing (ROME). We
find that ROME is effective on a standard zero-shot relation extraction (zsRE)
model-editing task, comparable to existing methods. To perform a more sensitive
evaluation, we also evaluate ROME on a new dataset of counterfactual
assertions, on which it simultaneously maintains both specificity and
generalization, whereas other methods sacrifice one or another. Our results
confirm an important role for mid-layer feed-forward modules in storing factual
associations and suggest that direct manipulation of computational mechanisms
may be a feasible approach for model editing. The code, dataset,
visualizations, and an interactive demo notebook are available at
this https URL

本文使用因果干预技术研究了自回归转换语言模型中实际关联的存储和检索，并发现这些关联对应于本地化的可直接编辑的计算。研究表明中间层前馈模块在存储实际关联方面具有重要作用，并且为模型编辑提供了直接操作计算机制的方法。