Although all-in-one-model multilingual neural machine translation
(multilingual NMT) has achieved remarkable progress, the convergence
inconsistency in the joint training is ignored, i.e., different language pairs
reaching convergence in different epochs. This leads to the trained MNMT model
over-fitting low-resource language translations while under-fitting
high-resource ones. In this paper, we propose a novel training strategy named
LSSD (Language-Specific Self-Distillation), which can alleviate the convergence
inconsistency and help MNMT models achieve the best performance on each
language pair simultaneously. Specifically, LSSD picks up language-specific
best checkpoints for each language pair to teach the current model on the fly.
Furthermore, we systematically explore three sample-level manipulations of
knowledge transferring. Experimental results on three datasets show that LSSD
obtains consistent improvements towards all language pairs and achieves the
state-of-the-art.

本文介绍了一种名为 LSSD（Language-Specific Self-Distillation）的新型训练策略，该策略可以缓解不同语言对之间的收敛不一致性并帮助 MNMT 模型同时在每种语言对上取得最佳性能。实验结果表明，LSSD 可以显著提高所有语言对的翻译性能，达到了最先进水平。