Multilingual neural machine translation has witnessed remarkable progress in
recent years. However, the long-tailed distribution of multilingual corpora
poses a challenge of Pareto optimization, i.e., optimizing for some languages
may come at the cost of degrading the performance of others. Existing balancing
training strategies are equivalent to a series of Pareto optimal solutions,
which trade off on a Pareto frontier. In this work, we propose a new training
framework, Pareto Mutual Distillation (Pareto-MD), towards pushing the Pareto
frontier outwards rather than making trade-offs. Specifically, Pareto-MD
collaboratively trains two Pareto optimal solutions that favor different
languages and allows them to learn from the strengths of each other via
knowledge distillation. Furthermore, we introduce a novel strategy to enable
stronger communication between Pareto optimal solutions and broaden the
applicability of our approach. Experimental results on the widely-used WMT and
TED datasets show that our method significantly pushes the Pareto frontier and
outperforms baselines by up to +2.46 BLEU.

本文提出了一种新的训练框架 Pareto Mutual Distillation (Pareto-MD)，通过知识蒸馏协同训练出针对不同语言的 Pareto 最优解，有力地推进了 Pareto 前沿，实验结果表明在 WMT 和 TED 数据集上，我们的方法使 Pareto 前沿显著提高，并且比基线模型的 BLEU 值提高了高达 2.46。