In real-world applications, dynamic scenarios require the models to possess
the capability to learn new tasks continuously without forgetting the old
knowledge. Experience-Replay methods store a subset of the old images for joint
training. In the scenario of more strict privacy protection, storing the old
images becomes infeasible, which leads to a more severe plasticity-stability
dilemma and classifier bias. To meet the above challenges, we propose a new
architecture, named continual expansion and absorption transformer~(CEAT). The
model can learn the novel knowledge by extending the expanded-fusion layers in
parallel with the frozen previous parameters. After the task ends, we
losslessly absorb the extended parameters into the backbone to ensure that the
number of parameters remains constant. To improve the learning ability of the
model, we designed a novel prototype contrastive loss to reduce the overlap
between old and new classes in the feature space. Besides, to address the
classifier bias towards the new classes, we propose a novel approach to
generate the pseudo-features to correct the classifier. We experiment with our
methods on three standard Non-Exemplar Class-Incremental Learning~(NECIL)
benchmarks. Extensive experiments demonstrate that our model gets a significant
improvement compared with the previous works and achieves 5.38%, 5.20%, and
4.92% improvement on CIFAR-100, TinyImageNet, and ImageNet-Subset.

通过延展融合层并与冻结先前参数并行，然后无损地将扩展参数吸收到主干网络中，以确保参数数量恒定，同时采用新型的原型对比损失减少特征空间中新旧类别的重叠，并提出一种生成伪特征以校正分类器的新方法。实验结果表明，在 CIFAR-100、TinyImageNet 和 ImageNet-Subset 上，与之前的研究相比，我们的模型获得了显著的改进，分别提高了 5.38％、5.20％和 4.92％。

CEAT：用于非示例类别增量学习的连续扩展和吸收变换器

CEAT: Continual Expansion and Absorption Transformer for Non-Exemplar  Class-Incremental Learnin

Plasticity-stability dilemma is a main problem for incremental learning,
where plasticity is referring to the ability to learn new knowledge, and
stability retains the knowledge of previous tasks. Many methods tackle this
problem by storing previous samples, while in some applications, training data
from previous tasks cannot be legally stored. In this work, we propose to
employ mode connectivity in loss landscapes to achieve better
plasticity-stability trade-off without any previous samples. We give an
analysis of why and how to connect two independently optimized optima of
networks, null-space projection for previous tasks and simple SGD for the
current task, can attain a meaningful balance between preserving already
learned knowledge and granting sufficient flexibility for learning a new task.
This analysis of mode connectivity also provides us a new perspective and
technology to control the trade-off between plasticity and stability. We
evaluate the proposed method on several benchmark datasets. The results
indicate our simple method can achieve notable improvement, and perform well on
both the past and current tasks. On 10-split-CIFAR-100 task, our method
achieves 79.79% accuracy, which is 6.02% higher. Our method also achieves 6.33%
higher accuracy on TinyImageNet. Code is available at
this https URL

本文章提出了一种基于模式连通性的损失地貌的方法，可以在不保留旧样本的情况下，实现更好的可塑性 - 稳定性权衡，该方法通过将两个独立优化的最优值连接起来实现一个有意义的平衡，我们在几个基准数据集上进行了评估，结果表明我们的简单方法可以提高模型性能。

增量学习中更好的可塑性 - 稳定性平衡：一种简单的线性连接器

Towards Better Plasticity-Stability Trade-off in Incremental Learning: A  Simple Linear Connector

Catastrophic forgetting affects the training of neural networks, limiting
their ability to learn multiple tasks sequentially. From the perspective of the
well established plasticity-stability dilemma, neural networks tend to be
overly plastic, lacking the stability necessary to prevent the forgetting of
previous knowledge, which means that as learning progresses, networks tend to
forget previously seen tasks. This phenomenon coined in the continual learning
literature, has attracted much attention lately, and several families of
approaches have been proposed with different degrees of success. However, there
has been limited prior work extensively analyzing the impact that different
training regimes -- learning rate, batch size, regularization method-- can have
on forgetting. In this work, we depart from the typical approach of altering
the learning algorithm to improve stability. Instead, we hypothesize that the
geometrical properties of the local minima found for each task play an
important role in the overall degree of forgetting. In particular, we study the
effect of dropout, learning rate decay, and batch size, on forming training
regimes that widen the tasks' local minima and consequently, on helping it not
to forget catastrophically. Our study provides practical insights to improve
stability via simple yet effective techniques that outperform alternative
baselines.

通过研究不同的训练规则，如 dropout、学习率衰减和 batch size 等，以形成训练模式，从而扩大任务的局部极小值，在帮助神经网络避免灾难性遗忘方面提供了实用的见解。