Neural networks tend to gradually forget the previously learned knowledge
when learning multiple tasks sequentially from dynamic data distributions. This
problem is called \textit{catastrophic forgetting}, which is a fundamental
challenge in the continual learning of neural networks. In this work, we
observe that catastrophic forgetting not only occurs in continual learning but
also affects the traditional static training. Neural networks, especially
neural machine translation models, suffer from catastrophic forgetting even if
they learn from a static training set. To be specific, the final model pays
imbalanced attention to training samples, where recently exposed samples
attract more attention than earlier samples. The underlying cause is that
training samples do not get balanced training in each model update, so we name
this problem \textit{imbalanced training}. To alleviate this problem, we
propose Complementary Online Knowledge Distillation (COKD), which uses
dynamically updated teacher models trained on specific data orders to
iteratively provide complementary knowledge to the student model. Experimental
results on multiple machine translation tasks show that our method successfully
alleviates the problem of imbalanced training and achieves substantial
improvements over strong baseline systems.

针对神经网络在从动态数据分布中连续学习多个任务时逐渐忘记以前学习的知识的问题，提出了一种补充在线知识蒸馏（COKD）的解决方法，成功地缓解了不平衡训练问题并在多个机器翻译任务上取得了实质性的改进。