Recent advances in training multilingual language models on large datasets seem to have shown promising results in knowledge transfer across languages and achieve high performance on downstream tasks. However, we question to what extent the current evaluation benchmarks and setups accurately measure zero-shot cross-lingual knowledge transfer. In this work, we challenge the assumption that high zero-shot performance on target tasks reflects high cross-lingual ability by introducing more challenging setups involving instances with multiple languages. Through extensive experiments and analysis, we show that the observed high performance of multilingual models can be largely attributed to factors not requiring the transfer of actual linguistic knowledge, such as task- and surface-level knowledge. More specifically, we observe what has been transferred across languages is mostly data artifacts and biases, especially for low-resource languages. Our findings highlight the overlooked drawbacks of existing cross-lingual test data and evaluation setups, calling for a more nuanced understanding of the cross-lingual capabilities of multilingual models.

通过实验证明多语言模型具有零-shot 跨语言知识转移的能力并且在下游任务中表现出较高性能，但我们对当前评估基准和设置是否能准确衡量零-shot 跨语言知识转移产生了质疑。本研究通过引入更具挑战性的多语言实例设置，表明多语言模型的高性能在很大程度上归因于不需要传递实际语言知识的因素，如任务和表面层知识。我们观察到跨语言传递的主要是数据工件和偏见，尤其是对于资源有限的语言。我们的发现凸显了现有跨语言测试数据和评估设置的缺点，呼吁对多语言模型的跨语言能力有更细致的理解。

多语言语言模型中跨语言知识迁移评估的分析