Continual learning (CL) has two main objectives: preventing catastrophic
forgetting (CF) and encouraging knowledge transfer (KT). The existing
literature mainly focused on overcoming CF. Some work has also been done on KT
when the tasks are similar. To our knowledge, only one method has been proposed
to learn a sequence of mixed tasks. However, these techniques still suffer from
CF and/or limited KT. This paper proposes a new CL method to achieve both. It
overcomes CF by isolating the knowledge of each task via discovering a
subnetwork for it. A soft-masking mechanism is also proposed to preserve the
previous knowledge and to enable the new task to leverage the past knowledge to
achieve KT. Experiments using classification, generation, information
extraction, and their mixture (i.e., heterogeneous tasks) show that the
proposed method consistently outperforms strong baselines.

本文提出了一种新的连续学习方法，通过发现为每个任务建立子网络来解决灾难性遗忘，同时引入了软掩模机制来保留先前的知识并实现知识传递，实验结果显示该方法始终优于强基线模型。