Addressing catastrophic forgetting is one of the key challenges in continual
learning where machine learning systems are trained with sequential or
streaming tasks. Despite recent remarkable progress in state-of-the-art deep
learning, deep neural networks (DNNs) are still plagued with the catastrophic
forgetting problem. This paper presents a conceptually simple yet general and
effective framework for handling catastrophic forgetting in continual learning
with DNNs. The proposed method consists of two components: a neural structure
optimization component and a parameter learning and/or fine-tuning component.
By separating the explicit neural structure learning and the parameter
estimation, not only is the proposed method capable of evolving neural
structures in an intuitively meaningful way, but also shows strong capabilities
of alleviating catastrophic forgetting in experiments. Furthermore, the
proposed method outperforms all other baselines on the permuted MNIST dataset,
the split CIFAR100 dataset and the Visual Domain Decathlon dataset in continual
learning setting.

本文提出了一个用于处理深度神经网络中的灾难性遗忘问题的概念简单、通用且有效的框架，通过优化神经结构和参数学习等两个组件，不仅可以直观有意义地演化神经结构，而且在实验证明了该方法具有很强的缓解灾难性遗忘问题的能力，此方法在连续学习的设置下，优于 MNIST、CIFAR100 和 Visual Domain Decathlon 数据集上的所有基线。