We study the underexplored problem of Continual Multilingual Learning, where
a multilingual model, already trained on task-specific data from all supported
languages, is continually updated using batches of new multilingual training
data for the same task. We show that naively updating the multilingual model
can lead to losses in performance over a subset of languages although the
aggregated performance metric shows an improvement. We establish this
phenomenon over four tasks belonging to three task families (token-level,
sentence-level and seq2seq). We then build upon recent advances in
parameter-efficient finetuning to develop novel finetuning strategies that
allow us to jointly minimize language-specific forgetting while encouraging
positive cross-lingual transfer observed in this setup. Our proposed pipeline,
LAFT-URIEL, improves the spread of gains over the supported languages while
reducing the magnitude of language-specific losses incurred.

研究 Continual Multilingual Learning 的问题，提出使用 parameter-efficient finetuning 更新 multilingual model 来减少语言特异性的遗忘，同时提高跨语言转移能力，通过 LAFT-URIEL pipeline 来改善支持语言的性能