Continual learning (CL) has shown promising results and comparable
performance to learning at once in a fully supervised manner. However, CL
strategies typically require a large number of labeled samples, making their
real-life deployment challenging. In this work, we focus on semi-supervised
continual learning (SSCL), where the model progressively learns from partially
labeled data with unknown categories. We provide a comprehensive analysis of
SSCL and demonstrate that unreliable distributions of unlabeled data lead to
unstable training and refinement of the progressing stages. This problem
severely impacts the performance of SSCL. To address the limitations, we
propose a novel approach called Dynamic Sub-Graph Distillation (DSGD) for
semi-supervised continual learning, which leverages both semantic and
structural information to achieve more stable knowledge distillation on
unlabeled data and exhibit robustness against distribution bias. Firstly, we
formalize a general model of structural distillation and design a dynamic graph
construction for the continual learning progress. Next, we define a structure
distillation vector and design a dynamic sub-graph distillation algorithm,
which enables end-to-end training and adaptability to scale up tasks. The
entire proposed method is adaptable to various CL methods and supervision
settings. Finally, experiments conducted on three datasets CIFAR10, CIFAR100,
and ImageNet-100, with varying supervision ratios, demonstrate the
effectiveness of our proposed approach in mitigating the catastrophic
forgetting problem in semi-supervised continual learning scenarios.

我们提出了一种名为动态子图蒸馏（DSGD）的新方法，它利用语义和结构信息，实现对无标签数据的稳定知识蒸馏，对分布偏差具有鲁棒性，并在半监督连续学习场景中减轻了灾难性遗忘问题。

动态子图提炼用于稳健的半监督连续学习

Dynamic Sub-graph Distillation for Robust Semi-supervised Continual  Learning

Continuous unsupervised representation learning (CURL) research has greatly
benefited from improvements in self-supervised learning (SSL) techniques. As a
result, existing CURL methods using SSL can learn high-quality representations
without any labels, but with a notable performance drop when learning on a
many-tasks data stream. We hypothesize that this is caused by the
regularization losses that are imposed to prevent forgetting, leading to a
suboptimal plasticity-stability trade-off: they either do not adapt fully to
the incoming data (low plasticity), or incur significant forgetting when
allowed to fully adapt to a new SSL pretext-task (low stability). In this work,
we propose to train an expert network that is relieved of the duty of keeping
the previous knowledge and can focus on performing optimally on the new tasks
(optimizing plasticity). In the second phase, we combine this new knowledge
with the previous network in an adaptation-retrospection phase to avoid
forgetting and initialize a new expert with the knowledge of the old network.
We perform several experiments showing that our proposed approach outperforms
other CURL exemplar-free methods in few- and many-task split settings.
Furthermore, we show how to adapt our approach to semi-supervised continual
learning (Semi-SCL) and show that we surpass the accuracy of other
exemplar-free Semi-SCL methods and reach the results of some others that use
exemplars.

通过训练专家网络和结合自适应回顾相位的方法来解决在多任务数据流上学习时的性能下降问题，该方法在少任务和多任务划分设置中优于其他无示例连续无监督表征学习方法，并且在半监督连续学习方面表现优于其他无示例半监督连续学习方法。