In the scenario of class-incremental learning (CIL), deep neural networks
have to adapt their model parameters to non-stationary data distributions,
e.g., the emergence of new classes over time. However, CIL models are
challenged by the well-known catastrophic forgetting phenomenon. Typical
methods such as rehearsal-based ones rely on storing exemplars of old classes
to mitigate catastrophic forgetting, which limits real-world applications
considering memory resources and privacy issues. In this paper, we propose a
novel rehearsal-free CIL approach that learns continually via the synergy
between two Complementary Learning Subnetworks. Our approach involves jointly
optimizing a plastic CNN feature extractor and an analytical feed-forward
classifier. The inaccessibility of historical data is tackled by holistically
controlling the parameters of a well-trained model, ensuring that the decision
boundary learned fits new classes while retaining recognition of previously
learned classes. Specifically, the trainable CNN feature extractor provides
task-dependent knowledge separately without interference; and the final
classifier integrates task-specific knowledge incrementally for decision-making
without forgetting. In each CIL session, it accommodates new tasks by attaching
a tiny set of declarative parameters to its backbone, in which only one matrix
per task or one vector per class is kept for knowledge retention. Extensive
experiments on a variety of task sequences show that our method achieves
competitive results against state-of-the-art methods, especially in accuracy
gain, memory cost, training efficiency, and task-order robustness. Furthermore,
to make the non-growing backbone (i.e., a model with limited network capacity)
suffice to train on more incoming tasks, a graceful forgetting implementation
on previously learned trivial tasks is empirically investigated.

本文提出了一种新的无需回放的类增量学习方法，该方法通过学习两个互补学习子网络之间的协同作用，包括联合优化可塑性 CNN 特征提取器和解析前馈分类器来实现。该方法在训练期间不需要存储过去的数据，可以掌握新的课程，同时保持对以前学习过的课程的识别和知识保留。

参数高效的增量式学习的互补学习子网络

Complementary Learning Subnetworks for Parameter-Efficient  Class-Incremental Learning

We introduce Continual Learning via Neural Pruning (CLNP), a new method aimed
at lifelong learning in fixed capacity models based on neuronal model
sparsification. In this method, subsequent tasks are trained using the inactive
neurons and filters of the sparsified network and cause zero deterioration to
the performance of previous tasks. In order to deal with the possible
compromise between model sparsity and performance, we formalize and incorporate
the concept of graceful forgetting: the idea that it is preferable to suffer a
small amount of forgetting in a controlled manner if it helps regain network
capacity and prevents uncontrolled loss of performance during the training of
future tasks. CLNP also provides simple continual learning diagnostic tools in
terms of the number of free neurons left for the training of future tasks as
well as the number of neurons that are being reused. In particular, we see in
experiments that CLNP verifies and automatically takes advantage of the fact
that the features of earlier layers are more transferable. We show empirically
that CLNP leads to significantly improved results over current weight
elasticity based methods.

本篇研究提出了一种名为 CLNP 的方法，通过神经模型稀疏化实现模型固定容量下的全生命周期学习，在这种方法中，使用经过稀疏化网络中的非活动神经元和滤波器来训练后续任务，并对以前任务的性能不会造成任何恶化，此外，CLNP 还提供了简单的学习诊断工具。实验证明，与当前基于权重弹性的方法相比，CLNP 能够显著提高结果。