Deep neural networks (DNNs) often suffer from "catastrophic forgetting"
during incremental learning (IL) --- an abrupt degradation of performance on
the original set of classes when the training objective is adapted to a newly
added set of classes. Existing IL approaches tend to produce a model that is
biased towards either the old classes or new classes, unless with the help of
exemplars of the old data. To address this issue, we propose a
class-incremental learning paradigm called Deep Model Consolidation (DMC),
which works well even when the original training data is not available. The
idea is to first train a separate model only for the new classes, and then
combine the two individual models trained on data of two distinct set of
classes (old classes and new classes) via a novel double distillation training
objective. The two existing models are consolidated by exploiting publicly
available unlabeled auxiliary data. This overcomes the potential difficulties
due to the unavailability of original training data. Compared to the
state-of-the-art techniques, DMC demonstrates significantly better performance
in image classification (CIFAR-100 and CUB-200) and object detection (PASCAL
VOC 2007) in the single-headed IL setting.

本文提出了一个名为 Deep Model Consolidation（DMC）的类增量学习范式，它通过利用公共可用的无标签辅助数据，将训练数据集中的已知类和未知类分别训练成不同的模型，然后通过新型的双重蒸馏训练目标结合两个模型，从而解决了增量学习中的 “灾难性遗忘” 问题，并在单头 IL 设置中，相比于现有技术，显著提升了图像分类和物体检测的性能。