Catastrophic forgetting and capacity saturation are the central challenges of
any parametric lifelong learning system. In this work, we study these
challenges in the context of sequential supervised learning with an emphasis on
recurrent neural networks. To evaluate the models in the lifelong learning
setting, we propose a curriculum-based, simple, and intuitive benchmark where
the models are trained on tasks with increasing levels of difficulty. To
measure the impact of catastrophic forgetting, the model is tested on all the
previous tasks as it completes any task. As a step towards developing true
lifelong learning systems, we unify Gradient Episodic Memory (a catastrophic
forgetting alleviation approach) and Net2Net(a capacity expansion approach).
Both these models are proposed in the context of feedforward networks and we
evaluate the feasibility of using them for recurrent networks. Evaluation on
the proposed benchmark shows that the unified model is more suitable than the
constituent models for lifelong learning setting.

本篇研究论文围绕序列监督学习中的参数化终身学习系统中的主要挑战 —— 灾难性遗忘和容量饱和进行了研究，并提出了一种基于课程的简单明了的评估标准，用于在不断增加难度的任务上对模型进行训练和测试。研究提出了两个网络模型 Gradient Episodic Memory 和 Net2Net，将它们统一起来，并将其应用于递归神经网络，验证了这种方法在终身学习环境中的适用性。