Catastrophic forgetting occurs when a neural network loses the information
learned in a previous task after training on subsequent tasks. This problem
remains a hurdle for artificial intelligence systems with sequential learning
capabilities. In this paper, we propose a task-based hard attention mechanism
that preserves previous tasks' information without affecting the current task's
learning. A hard attention mask is learned concurrently to every task, through
stochastic gradient descent, and previous masks are exploited to condition such
learning. We show that the proposed mechanism is effective for reducing
catastrophic forgetting, cutting current rates by 45 to 80%. We also show that
it is robust to different hyperparameter choices, and that it offers a number
of monitoring capabilities. The approach features the possibility to control
both the stability and compactness of the learned knowledge, which we believe
makes it also attractive for online learning or network compression
applications.

本文提出了一种基于任务的硬注意机制，利用随机梯度下降学习硬注意掩码来保留上一个任务中的信息而不影响当前任务的学习，有效提高神经网络在顺序学习能力上的性能。此方法对不同超参数的选择也具有鲁棒性，并且具有控制学习知识稳定性和紧凑性等特点，在在线学习或网络压缩应用中也是吸引人的。