Multilingual machine translation (MMT) benefits from cross-lingual transfer
but is a challenging multitask optimization problem. This is partly because
there is no clear framework to systematically learn language-specific
parameters. Self-supervised learning (SSL) approaches that leverage large
quantities of monolingual data (where parallel data is unavailable) have shown
promise by improving translation performance as complementary tasks to the MMT
task. However, jointly optimizing SSL and MMT tasks is even more challenging.
In this work, we first investigate how to utilize intra-distillation to learn
more *language-specific* parameters and then show the importance of these
language-specific parameters. Next, we propose a novel but simple SSL task,
concurrent denoising, that co-trains with the MMT task by concurrently
denoising monolingual data on both the encoder and decoder. Finally, we apply
intra-distillation to this co-training approach. Combining these two approaches
significantly improves MMT performance, outperforming three state-of-the-art
SSL methods by a large margin, e.g., 11.3\% and 3.7\% improvement on an
8-language and a 15-language benchmark compared with MASS, respectively

本文探讨了如何利用自监督学习的方法，在多语言机器翻译任务中实现参数的有利学习，其中采用了基于噪声消除的简单自监督任务，并将其与原有任务同时进行。通过这两种方法的结合，在 8 种和 15 种语言基准测试中分别比 MASS 等三种最先进的自监督学习方法提高了 11.3％和 3.7％的性能。

具备语言意识的自监督学习多语言机器翻译

Language-Aware Multilingual Machine Translation with Self-Supervised Learning

Recent model pruning methods have demonstrated the ability to remove
redundant parameters without sacrificing model performance. Common methods
remove redundant parameters according to the parameter sensitivity, a
gradient-based measure reflecting the contribution of the parameters. In this
paper, however, we argue that redundant parameters can be trained to make
beneficial contributions. We first highlight the large sensitivity
(contribution) gap among high-sensitivity and low-sensitivity parameters and
show that the model generalization performance can be significantly improved
after balancing the contribution of all parameters. Our goal is to balance the
sensitivity of all parameters and encourage all of them to contribute equally.
We propose a general task-agnostic method, namely intra-distillation, appended
to the regular training loss to balance parameter sensitivity. Moreover, we
also design a novel adaptive learning method to control the strength of
intra-distillation loss for faster convergence. Our experiments show the strong
effectiveness of our methods on machine translation, natural language
understanding, and zero-shot cross-lingual transfer across up to 48 languages,
e.g., a gain of 3.54 BLEU on average across 8 language pairs from the IWSLT'14
translation dataset.

本文提出了一种基于 intra-distillation 和自适应学习的通用方法，用于平衡所有参数的灵敏度，从而提高模型的普适性和性能表现。实验结果表明，该方法有效地提升了机器翻译、自然语言理解和零 - shot 跨语言转移等不同领域的表现。