Incremental learning (IL) is an important task aimed at increasing the
capability of a trained model, in terms of the number of classes recognizable
by the model. The key problem in this task is the requirement of storing data
(e.g. images) associated with existing classes, while teaching the classifier
to learn new classes. However, this is impractical as it increases the memory
requirement at every incremental step, which makes it impossible to implement
IL algorithms on edge devices with limited memory. Hence, we propose a novel
approach, called `Learning without Memorizing (LwM)', to preserve the
information about existing (base) classes, without storing any of their data,
while making the classifier progressively learn the new classes. In LwM, we
present an information preserving penalty: Attention Distillation Loss
($L_{AD}$), and demonstrate that penalizing the changes in classifiers'
attention maps helps to retain information of the base classes, as new classes
are added. We show that adding $L_{AD}$ to the distillation loss which is an
existing information preserving loss consistently outperforms the
state-of-the-art performance in the iILSVRC-small and iCIFAR-100 datasets in
terms of the overall accuracy of base and incrementally learned classes.

提出一种新方法 “无需记忆学习 (LwM)”，通过信息保留惩罚方法 Attention Distillation Loss (L_AD) 来保存基础类的信息，同时逐步学习新类，实现在设备内存有限的情况下进行增量学习并取得更好的效果，实验证明在 iILSVRC-small 和 iCIFAR-100 数据集上，加入 L_AD 相较于现有的信息保留方法有更好的基础和增量学习准确度。