Continual learning (CL) is one of the most promising trends in recent machine
learning research. Its goal is to go beyond classical assumptions in machine
learning and develop models and learning strategies that present high
robustness in dynamic environments. The landscape of CL research is fragmented
into several learning evaluation protocols, comprising different learning
tasks, datasets, and evaluation metrics. Additionally, the benchmarks adopted
so far are still distant from the complexity of real-world scenarios, and are
usually tailored to highlight capabilities specific to certain strategies. In
such a landscape, it is hard to objectively assess strategies. In this work, we
fill this gap for CL on image data by introducing two novel CL benchmarks that
involve multiple heterogeneous tasks from six image datasets, with varying
levels of complexity and quality. Our aim is to fairly evaluate current
state-of-the-art CL strategies on a common ground that is closer to complex
real-world scenarios. We additionally structure our benchmarks so that tasks
are presented in increasing and decreasing order of complexity -- according to
a curriculum -- in order to evaluate if current CL models are able to exploit
structure across tasks. We devote particular emphasis to providing the CL
community with a rigorous and reproducible evaluation protocol for measuring
the ability of a model to generalize and not to forget while learning.
Furthermore, we provide an extensive experimental evaluation showing that
popular CL strategies, when challenged with our benchmarks, yield sub-par
performance, high levels of forgetting, and present a limited ability to
effectively leverage curriculum task ordering. We believe that these results
highlight the need for rigorous comparisons in future CL works as well as pave
the way to design new CL strategies that are able to deal with more complex
scenarios.

本研究针对机器学习中的不断学习提出了两个新的基准，该基准涉及来自六个图像数据集的多个异构任务，其目的是为了更好地评估当前最先进的 CL 策略，并显示出当前 CL 模型在真实世界场景中表现较差的能力，高水平遗忘并限制了课程任务顺序。