Modular deep learning is the state-of-the-art solution for lifting the curse
of multilinguality, preventing the impact of negative interference and enabling
cross-lingual performance in Multilingual Pre-trained Language Models. However,
a trade-off of this approach is the reduction in positive transfer learning
from closely related languages. In response, we introduce a novel method called
language arithmetic, which enables training-free post-processing to address
this limitation. Inspired by the task arithmetic framework, we apply learning
via addition to the language adapters, transitioning the framework from a
multi-task to a multilingual setup. The effectiveness of the proposed solution
is demonstrated on three downstream tasks in a MAD-X-based set of cross-lingual
schemes, acting as a post-processing procedure. Language arithmetic
consistently improves the baselines with significant gains in the most
challenging cases of zero-shot and low-resource applications. Our code and
models are available at this https URL .

在多语言预训练语言模型中，模块化深度学习是解决多语言性的技术，而语言算术是一种训练后的后处理方法，能够显著提升在零样本和低资源应用中的多语言性能。

无训练即获益：用于无需训练的语言适配器增强的语言算术

No Train but Gain: Language Arithmetic for training-free Language  Adapters enhancement

The rise of Modular Deep Learning showcases its potential in various Natural
Language Processing applications. Parameter-efficient fine-tuning (PEFT)
modularity has been shown to work for various use cases, from domain adaptation
to multilingual setups. However, all this work covers the case where the
modular components are trained and deployed within one single Pre-trained
Language Model (PLM). This model-specific setup is a substantial limitation on
the very modularity that modular architectures are trying to achieve. We ask
whether current modular approaches are transferable between models and whether
we can transfer the modules from more robust and larger PLMs to smaller ones.
In this work, we aim to fill this gap via a lens of Knowledge Distillation,
commonly used for model compression, and present an extremely straightforward
approach to transferring pre-trained, task-specific PEFT modules between
same-family PLMs. Moreover, we propose a method that allows the transfer of
modules between incompatible PLMs without any change in the inference
complexity. The experiments on Named Entity Recognition, Natural Language
Inference, and Paraphrase Identification tasks over multiple languages and PEFT
methods showcase the initial potential of transferable modularity.

模块化深度学习在自然语言处理应用中展现了潜力，而参数高效微调（PEFT）模块化已经表明其适用于各种情况，从领域自适应到多语言设置。本文通过知识蒸馏的视角填补了现有模块化方法的一个重要空白，并提出了一种非常简单的方法，在同一家族的预训练语言模型（PLM）之间传递预训练、任务特定的 PEFT 模块。此外，我们提出了一种允许在不改变推理复杂度的情况下在不兼容的 PLM 之间传递模块的方法。多语言和 PEFT 方法上对命名实体识别、自然语言推理和剽窃识别任务的实验展示了可传递的模块化的初步潜力。

模块性是否可转移？通过知识蒸馏的案例研究

Is Modularity Transferable? A Case Study through the Lens of Knowledge  Distillation

Modular deep learning has been proposed for the efficient adaption of
pre-trained models to new tasks, domains and languages. In particular,
combining language adapters with task adapters has shown potential where no
supervised data exists for a language. In this paper, we explore the role of
language adapters in zero-shot cross-lingual transfer for natural language
understanding (NLU) benchmarks. We study the effect of including a
target-language adapter in detailed ablation studies with two multilingual
models and three multilingual datasets. Our results show that the effect of
target-language adapters is highly inconsistent across tasks, languages and
models. Retaining the source-language adapter instead often leads to an
equivalent, and sometimes to a better, performance. Removing the language
adapter after training has only a weak negative effect, indicating that the
language adapters do not have a strong impact on the predictions.

在不存在语言的监督数据的情况下，本文研究了语言适配器在自然语言理解任务中零样本跨语言转移中的作用，通过对两个多语言模型和三个多语言数据集的详尽消融研究，结果表明目标语言适配器在任务、语言和模型间影响高度不一致。相比之下，保留源语言适配器通常能够获得相当甚至更好的性能，训练后去除语言适配器只有较弱的负面影响，表明语言适配器对于预测结果的影响并不显著。