Continual learning, involving sequential training on diverse tasks, often
faces catastrophic forgetting. While knowledge distillation-based approaches
exhibit notable success in preventing forgetting, we pinpoint a limitation in
their ability to distill the cumulative knowledge of all the previous tasks. To
remedy this, we propose Dense Knowledge Distillation (DKD). DKD uses a task
pool to track the model's capabilities. It partitions the output logits of the
model into dense groups, each corresponding to a task in the task pool. It then
distills all tasks' knowledge using all groups. However, using all the groups
can be computationally expensive, we also suggest random group selection in
each optimization step. Moreover, we propose an adaptive weighting scheme,
which balances the learning of new classes and the retention of old classes,
based on the count and similarity of the classes. Our DKD outperforms recent
state-of-the-art baselines across diverse benchmarks and scenarios. Empirical
analysis underscores DKD's ability to enhance model stability, promote flatter
minima for improved generalization, and remains robust across various memory
budgets and task orders. Moreover, it seamlessly integrates with other CL
methods to boost performance and proves versatile in offline scenarios like
model compression.

连续学习面临的致命遗忘问题可以通过稠密知识蒸馏方法得到改善，该方法可以在所有任务间蒸馏累积知识，提高模型的稳定性。