The aim of continual learning is to learn new tasks continuously (i.e.,
plasticity) without forgetting previously learned knowledge from old tasks
(i.e., stability). In the scenario of online continual learning, wherein data
comes strictly in a streaming manner, the plasticity of online continual
learning is more vulnerable than offline continual learning because the
training signal that can be obtained from a single data point is limited. To
overcome the stability-plasticity dilemma in online continual learning, we
propose an online continual learning framework named multi-scale feature
adaptation network (MuFAN) that utilizes a richer context encoding extracted
from different levels of a pre-trained network. Additionally, we introduce a
novel structure-wise distillation loss and replace the commonly used batch
normalization layer with a newly proposed stability-plasticity normalization
module to train MuFAN that simultaneously maintains high plasticity and
stability. MuFAN outperforms other state-of-the-art continual learning methods
on the SVHN, CIFAR100, miniImageNet, and CORe50 datasets. Extensive experiments
and ablation studies validate the significance and scalability of each proposed
component: 1) multi-scale feature maps from a pre-trained encoder, 2) the
structure-wise distillation loss, and 3) the stability-plasticity normalization
module in MuFAN. Code is publicly available at
this https URL.

本研究提出了一种名为 MuFAN 的在线连续学习框架，利用不同级别的预训练网络中提取的更丰富的上下文编码，并引入了一种新颖的基于结构的蒸馏损失和稳定性 - 可塑性标准化模块，用于同时保持高可塑性和稳定性。MuFAN 在多个数据集上优于其他现有的连续学习方法。