Current evaluations of Continual Learning (CL) methods typically assume that
there is no constraint on training time and computation. This is an unrealistic
assumption for any real-world setting, which motivates us to propose: a
practical real-time evaluation of continual learning, in which the stream does
not wait for the model to complete training before revealing the next data for
predictions. To do this, we evaluate current CL methods with respect to their
computational costs. We conduct extensive experiments on CLOC, a large-scale
dataset containing 39 million time-stamped images with geolocation labels. We
show that a simple baseline outperforms state-of-the-art CL methods under this
evaluation, questioning the applicability of existing methods in realistic
settings. In addition, we explore various CL components commonly used in the
literature, including memory sampling strategies and regularization approaches.
We find that all considered methods fail to be competitive against our simple
baseline. This surprisingly suggests that the majority of existing CL
literature is tailored to a specific class of streams that is not practical. We
hope that the evaluation we provide will be the first step towards a paradigm
shift to consider the computational cost in the development of online continual
learning methods.

该研究提出了一种实用的、实时的、基于计算成本的 Continual Learning 方法评估方法，并使用包含 3900 万个时间戳标记的图像的 CLOC 数据集展开了大量实验，发现现有的 CL 方法并不适用于现实场景，并且介绍了记忆取样策略和正则化方法等常用 CL 组件，证实了对计算成本进行考虑是开发在线 Continual Learning 方法的必要基础。