Online Continual Learning (OCL) addresses the problem of training neural
networks on a continuous data stream where multiple classification tasks emerge
in sequence. In contrast to offline Continual Learning, data can be seen only
once in OCL. In this context, replay-based strategies have achieved impressive
results and most state-of-the-art approaches are heavily depending on them.
While Knowledge Distillation (KD) has been extensively used in offline
Continual Learning, it remains under-exploited in OCL, despite its potential.
In this paper, we theoretically analyze the challenges in applying KD to OCL.
We introduce a direct yet effective methodology for applying Momentum Knowledge
Distillation (MKD) to many flagship OCL methods and demonstrate its
capabilities to enhance existing approaches. In addition to improving existing
state-of-the-arts accuracy by more than $10\%$ points on ImageNet100, we shed
light on MKD internal mechanics and impacts during training in OCL. We argue
that similar to replay, MKD should be considered a central component of OCL.

应用动量知识蒸馏（MKD）来加强在线连续学习（OCL）方法，提高现有成果的准确度，将 MKD 视为 OCL 的核心组成部分。