The capacity and effectiveness of pre-trained multilingual models (MLMs) for
zero-shot cross-lingual transfer is well established. However, phenomena of
positive or negative transfer, and the effect of language choice still need to
be fully understood, especially in the complex setting of massively
multilingual LMs. We propose an \textit{efficient} method to study transfer
language influence in zero-shot performance on another target language. Unlike
previous work, our approach disentangles downstream tasks from language, using
dedicated adapter units. Our findings suggest that some languages do not
largely affect others, while some languages, especially ones unseen during
pre-training, can be extremely beneficial or detrimental for different target
languages. We find that no transfer language is beneficial for all target
languages. We do, curiously, observe languages previously unseen by MLMs
consistently benefit from transfer from almost any language. We additionally
use our modular approach to quantify negative interference efficiently and
categorize languages accordingly. Furthermore, we provide a list of promising
transfer-target language configurations that consistently lead to target
language performance improvements. Code and data are publicly available:
this https URL

预训练多语言模型的容量和效果已经得到确认，但对于零样本跨语言转移中的积极或消极转移现象以及语言选择的影响还需进一步理解，本研究提出了一种高效的方法，通过专用适配器单元将下游任务与语言分离，发现一些语言对其他语言影响不大，而一些未在预训练中出现的语言对不同目标语言具有极大益处或有害，我们发现没有任何一种语言对所有目标语言都有益，但奇怪的是我们观察到，之前未被多语言模型预训练见过的语言总是从任何语言的转移中受益，此外，我们利用模块化方法高效量化负面干涉并相应分类语言，最后，我们提供了一系列有希望改善目标语言性能的转移 - 目标语言配置。

多语言语言模型中研究跨语言传递的高效方法

An Efficient Approach for Studying Cross-Lingual Transfer in  Multilingual Language Models

We consider the task of optimally fine-tuning pre-trained multilingual
models, given small amounts of unlabelled target data and an annotation budget.
In this paper, we introduce DEMUX, a framework that prescribes the exact
data-points to label from vast amounts of unlabelled multilingual data, having
unknown degrees of overlap with the target set. Unlike most prior works, our
end-to-end framework is language-agnostic, accounts for model representations,
and supports multilingual target configurations. Our active learning strategies
rely upon distance and uncertainty measures to select task-specific neighbors
that are most informative to label, given a model. DeMuX outperforms strong
baselines in 84% of the test cases, in the zero-shot setting of disjoint source
and target language sets (including multilingual target pools), across three
models and four tasks. Notably, in low-budget settings (5-100 examples), we
observe gains of up to 8-11 F1 points for token-level tasks, and 2-5 F1 for
complex tasks. Our code is released here:
this https URL

优化预训练的多语言模型，在有限的目标数据和注释预算的情况下，使用 DEMUX 框架进行精确的标注点选择，通过距离和不确定性措施选取最具信息量的任务相关邻居，取得优于强基线的效果，特别是在低预算情况下可提高 F1 分数。