Instance-incremental learning (IIL) focuses on learning continually with data
of the same classes. Compared to class-incremental learning (CIL), the IIL is
seldom explored because IIL suffers less from catastrophic forgetting (CF).
However, besides retaining knowledge, in real-world deployment scenarios where
the class space is always predefined, continual and cost-effective model
promotion with the potential unavailability of previous data is a more
essential demand. Therefore, we first define a new and more practical IIL
setting as promoting the model's performance besides resisting CF with only new
observations. Two issues have to be tackled in the new IIL setting: 1) the
notorious catastrophic forgetting because of no access to old data, and 2)
broadening the existing decision boundary to new observations because of
concept drift. To tackle these problems, our key insight is to moderately
broaden the decision boundary to fail cases while retain old boundary. Hence,
we propose a novel decision boundary-aware distillation method with
consolidating knowledge to teacher to ease the student learning new knowledge.
We also establish the benchmarks on existing datasets Cifar-100 and ImageNet.
Notably, extensive experiments demonstrate that the teacher model can be a
better incremental learner than the student model, which overturns previous
knowledge distillation-based methods treating student as the main role.

在实例增量学习中，我们定义了一种新的实用模型推广设置，通过仅使用新观察来提高模型性能，同时抵制灾难性遗忘和应对概念漂移问题，我们提出了一种新颖的决策边界感知蒸馏方法来整理知识，使教师模型成为更好的增量学习器，这颠覆了之前以学生模型为主要角色的知识蒸馏方法。

决策边界感知的知识整合产生更好的增量学习器

Decision Boundary-aware Knowledge Consolidation Generates Better  Instance-Incremental Learner

Incremental learning (IL) is essential to realize the human-level
intelligence in the neural network. However, existing IL scenarios and datasets
are unqualified for assessing forgetting in PLMs, giving an illusion that PLMs
do not suffer from catastrophic forgetting. To this end, we propose a
challenging IL scenario called instance-incremental learning (IIL) and a novel
dataset called Concept-1K, which supports an order of magnitude larger IL
steps. Based on the experiments on Concept-1K, we reveal that billion-parameter
PLMs still suffer from catastrophic forgetting, and the forgetting is affected
by both model scale, pretraining, and buffer size. Furthermore, existing IL
methods and a popular finetuning technique, LoRA, fail to achieve satisfactory
performance. Our study provides a novel scenario for future studies to explore
the catastrophic forgetting of PLMs and encourage more powerful techniques to
be designed for alleviating the forgetting in PLMs. The data, code and scripts
are publicly available at
this https URL

基于对 Concept-1K 实验的研究，揭示了数十亿参数的预训练语言模型仍然受到灾难性遗忘的影响，并且遗忘受模型规模、预训练和缓冲区大小的影响。同时，现有的增量学习方法和一种流行的微调技术 LoRA 都无法达到令人满意的性能。这项研究为进一步探索预训练语言模型的灾难性遗忘提供了新的研究场景，鼓励设计更强大的技术来缓解预训练语言模型的遗忘问题。