Deep learning systems are prone to catastrophic forgetting when learning from
a sequence of tasks, where old data from experienced tasks is unavailable when
learning from a new task. To mitigate the problem, a line of methods propose to
replay the data of experienced tasks when learning new tasks. These methods
usually adopt an extra memory to store the data for replay. However, it is not
expected in practice considering the memory constraint or data privacy issue.
As a replacement, data-free data replay methods are proposed by inverting
samples from the classification model. Though achieving good results, these
methods still suffer from the inconsistency of the inverted and real training
data, which is neglected in the inversion stage in recent works. To that
effect, we propose to measure the data consistency quantitatively by some
simplification and assumptions. Using the measurement, we analyze existing
techniques for inverting samples and get some insightful information that
inspires a novel loss function to reduce the inconsistency. Specifically, the
loss minimizes the KL divergence of the distributions of inverted and real data
under the tied multivariate Gaussian assumption, which is easy to implement in
continual learning. In addition, we observe that the norms of old class weights
turn to decrease continually as learning progresses. We thus analyze the
underlying reasons and propose a simple regularization term to balance the
class weights so that the samples of old classes are more distinguishable. To
conclude, we propose the Consistency enhanced data replay with debiased
classifier for Class Incremental Learning (CCIL). Extensive experiments on
CIFAR-100, Tiny-ImageNet, and ImageNet100 show consistently improved
performance of CCIL compared to previous approaches.

提出了一种通过量化数据一致性并引入新的损失函数来改进数据回放及类别增量学习的方法。在实验证明比以前的方法有更好的性能。