Real-life multilingual systems should be able to efficiently incorporate new
languages as data distributions fed to the system evolve and shift over time.
To do this, systems need to handle the issue of catastrophic forgetting, where
the model performance drops for languages or tasks seen further in its past. In
this paper, we study catastrophic forgetting, as well as methods to minimize
this, in a massively multilingual continual learning framework involving up to
51 languages and covering both classification and sequence labeling tasks. We
present LR ADJUST, a learning rate scheduling method that is simple, yet
effective in preserving new information without strongly overwriting past
knowledge. Furthermore, we show that this method is effective across multiple
continual learning approaches. Finally, we provide further insights into the
dynamics of catastrophic forgetting in this massively multilingual setup.

本论文研究在一种包含 51 种语言和分类、序列标记任务的大规模多语言持续学习框架中，灾难性遗忘问题以及最小化此问题的方法，提出了一种学习率调整方法 LR ADJUST，其能有效地保留新信息而不过度覆盖过去的知识，并且该方法在多种持续学习方法中都行之有效。还进一步深入分析了该大规模多语言环境下灾难性遗忘的动态过程。