Continual learning (CL) is widely regarded as crucial challenge for lifelong
AI. However, existing CL benchmarks, e.g. Permuted-MNIST and Split-CIFAR, make
use of artificial temporal variation and do not align with or generalize to the
real-world. In this paper, we introduce CLEAR, the first continual image
classification benchmark dataset with a natural temporal evolution of visual
concepts in the real world that spans a decade (2004-2014). We build CLEAR from
existing large-scale image collections (YFCC100M) through a novel and scalable
low-cost approach to visio-linguistic dataset curation. Our pipeline makes use
of pretrained vision-language models (e.g. CLIP) to interactively build labeled
datasets, which are further validated with crowd-sourcing to remove errors and
even inappropriate images (hidden in original YFCC100M). The major strength of
CLEAR over prior CL benchmarks is the smooth temporal evolution of visual
concepts with real-world imagery, including both high-quality labeled data
along with abundant unlabeled samples per time period for continual
semi-supervised learning. We find that a simple unsupervised pre-training step
can already boost state-of-the-art CL algorithms that only utilize
fully-supervised data. Our analysis also reveals that mainstream CL evaluation
protocols that train and test on iid data artificially inflate performance of
CL system. To address this, we propose novel "streaming" protocols for CL that
always test on the (near) future. Interestingly, streaming protocols (a) can
simplify dataset curation since today's testset can be repurposed for
tomorrow's trainset and (b) can produce more generalizable models with more
accurate estimates of performance since all labeled data from each time-period
is used for both training and testing (unlike classic iid train-test splits).

本文介绍了 CLEAR 数据集，通过一种新颖的、可扩展的、低成本的视觉语言数据集策划方法，利用预训练的视觉语言模型 CLIP 交互式地构建标记数据集并进行验证，清除了错误和不合适的图像，提出了流处理评估协议并发现，流处理协议可以产生模型更普遍的、更准确的性能估计，包括半监督学习。