Incorporating language-specific (LS) modules is a proven method to boost
performance in multilingual machine translation. This approach bears similarity
to Mixture-of-Experts (MoE) because it does not inflate FLOPs. However, the
scalability of this approach to hundreds of languages (experts) tends to be
unmanageable due to the prohibitive number of parameters introduced by
full-rank matrices in fully-connected layers. In this work, we introduce the
Language-Specific Matrix Synthesis (LMS) method. This approach constructs LS
modules by generating low-rank matrices from two significantly smaller matrices
to approximate the full-rank matrix. Furthermore, we condense multilingual
knowledge from multiple LS modules into a single shared module with the Fuse
Distillation (FD) technique to improve the efficiency of inference and model
serialization. We show that our LMS method significantly outperforms previous
LS methods and MoE methods with the same amount of extra parameters, e.g., 1.73
BLEU points over the Switch Transformer on many-to-many multilingual machine
translation. Importantly, LMS is able to have comparable translation
performance with much fewer parameters.

本文提出了 Language-Specific Matrix Synthesis (LMS) 方法以解决多语言机器翻译领域中语言特定模块的可扩展性问题，并通过 Fuse Distillation 技术将模块知识压缩至单个共享模块提高推理效率和模型序列化能力。该方法在与同样额外参数的 LS 方法和 MoE 方法相比中获得更好的翻译性能，而且有着更少的参数。

使用轻量级的语言特定模块压缩多语言知识

Condensing Multilingual Knowledge with Lightweight Language-Specific  Modules

Multilingual pre-trained models are known to suffer from the curse of
multilinguality, which causes per-language performance to drop as they cover
more languages. We address this issue by introducing language-specific modules,
which allows us to grow the total capacity of the model, while keeping the
total number of trainable parameters per language constant. In contrast with
prior work that learns language-specific components post-hoc, we pre-train the
modules of our Cross-lingual Modular (X-Mod) models from the start. Our
experiments on natural language inference, named entity recognition and
question answering show that our approach not only mitigates the negative
interference between languages, but also enables positive transfer, resulting
in improved monolingual and cross-lingual performance. Furthermore, our
approach enables adding languages post-hoc with no measurable drop in
performance, no longer limiting the model usage to the set of pre-trained
languages.

该研究通过引入语言特定模块解决了多语言预训练模型中语言性能下降的问题，并在自然语言推断，命名实体识别和问答等方面证明了该方法的有效性，同时也使得可以在不降低性能的情况下添加新的语言。