Instance-incremental learning (IIL) focuses on learning continually with data
of the same classes. Compared to class-incremental learning (CIL), the IIL is
seldom explored because IIL suffers less from catastrophic forgetting (CF).
However, besides retaining knowledge, in real-world deployment scenarios where
the class space is always predefined, continual and cost-effective model
promotion with the potential unavailability of previous data is a more
essential demand. Therefore, we first define a new and more practical IIL
setting as promoting the model's performance besides resisting CF with only new
observations. Two issues have to be tackled in the new IIL setting: 1) the
notorious catastrophic forgetting because of no access to old data, and 2)
broadening the existing decision boundary to new observations because of
concept drift. To tackle these problems, our key insight is to moderately
broaden the decision boundary to fail cases while retain old boundary. Hence,
we propose a novel decision boundary-aware distillation method with
consolidating knowledge to teacher to ease the student learning new knowledge.
We also establish the benchmarks on existing datasets Cifar-100 and ImageNet.
Notably, extensive experiments demonstrate that the teacher model can be a
better incremental learner than the student model, which overturns previous
knowledge distillation-based methods treating student as the main role.

在实例增量学习中，我们定义了一种新的实用模型推广设置，通过仅使用新观察来提高模型性能，同时抵制灾难性遗忘和应对概念漂移问题，我们提出了一种新颖的决策边界感知蒸馏方法来整理知识，使教师模型成为更好的增量学习器，这颠覆了之前以学生模型为主要角色的知识蒸馏方法。