Real-world applications often require learning continuously from a stream of data under ever-changing conditions. When trying to learn from such non-stationary data, deep neural networks (DNNs) undergo catastrophic forgetting of previously learned information. Among the common approaches to avoid catastrophic forgetting, rehearsal-based methods have proven effective. However, they are still prone to forgetting due to task-interference as all parameters respond to all tasks. To counter this, we take inspiration from sparse coding in the brain and introduce dynamic modularity and sparsity (Dynamos) for rehearsal-based general continual learning. In this setup, the DNN learns to respond to stimuli by activating relevant subsets of neurons. We demonstrate the effectiveness of Dynamos on multiple datasets under challenging continual learning evaluation protocols. Finally, we show that our method learns representations that are modular and specialized, while maintaining reusability by activating subsets of neurons with overlaps corresponding to the similarity of stimuli.

本文研究了基于重复练习的持续学习方法对于解决深度神经网络在面对非稳态数据时产生的遗忘问题的有效性，并引入 brain sparse coding 中的动态模块化与稀疏性 (Dynamos) 来解决任务干扰问题。研究结果表明 Dynamos 在多组数据集上的表现均好，并且所学习的特征表现出模块化和专业化的特点，同时具有重用性。

动态模块化和稀疏的通用持续学习