We consider class incremental learning (CIL) problem, in which a learning
agent continuously learns new classes from incrementally arriving training data
batches and aims to predict well on all the classes learned so far. The main
challenge of the problem is the catastrophic forgetting, and for the
exemplar-memory based CIL methods, it is generally known that the forgetting is
commonly caused by the classification score bias that is injected due to the
data imbalance between the new classes and the old classes (in the
exemplar-memory). While several methods have been proposed to correct such
score bias by some additional post-processing, e.g., score re-scaling or
balanced fine-tuning, no systematic analysis on the root cause of such bias has
been done. To that end, we analyze that computing the softmax probabilities by
combining the output scores for all old and new classes could be the main cause
of the bias. Then, we propose a new method, dubbed as Separated Softmax for
Incremental Learning (SS-IL), that consists of separated softmax (SS) output
layer combined with task-wise knowledge distillation (TKD) to resolve such
bias. Throughout our extensive experimental results on several large-scale CIL
benchmark datasets, we show our SS-IL achieves strong state-of-the-art accuracy
through attaining much more balanced prediction scores across old and new
classes, without any additional post-processing.

通过对先前流派中分类分数导致偏差的原因进行系统分析，我们提出了一种称为 Separate softmax for incremental learning (SS-IL) 的新方法，它由分离的 softmax (SS) 输出层和面向任务的知识蒸馏 (TKD) 组成，以解决诸如数据失衡等问题，从而在多个大规模 CIL 基准数据集上实现了强大的最新结果。