Catastrophic forgetting in neural networks is a significant problem for
continual learning. A majority of the current methods replay previous data
during training, which violates the constraints of an ideal continual learning
system. Additionally, current approaches that deal with forgetting ignore the
problem of catastrophic remembering, i.e. the worsening ability to discriminate
between data from different tasks. In our work, we introduce Relevance Mapping
Networks (RMNs) which are inspired by the Optimal Overlap Hypothesis. The
mappings reflects the relevance of the weights for the task at hand by
assigning large weights to essential parameters. We show that RMNs learn an
optimized representational overlap that overcomes the twin problem of
catastrophic forgetting and remembering. Our approach achieves state-of-the-art
performance across all common continual learning datasets, even significantly
outperforming data replay methods while not violating the constraints for an
ideal continual learning system. Moreover, RMNs retain the ability to detect
data from new tasks in an unsupervised manner, thus proving their resilience
against catastrophic remembering.

本文介绍了一种基于相关性映射网络 (RMNs) 的新方法，来解决神经网络模型中的 “灾难性遗忘” 问题，该方法受到最优重叠假设的启发，并能够在不违反理想的连续学习系统约束的情况下，显着超越数据重放方法，成为最优连续学习数据集中的主流方法；同时，RMNs 还具备检测新任务数据的无监督学习能力。