Current state-of-the-art vision-and-language models are evaluated on tasks
either individually or in a multi-task setting, overlooking the challenges of
continually learning (CL) tasks as they arrive. Existing CL benchmarks have
facilitated research on task adaptation and mitigating "catastrophic
forgetting", but are limited to vision-only and language-only tasks. We present
CLiMB, a benchmark to study the challenge of learning multimodal tasks in a CL
setting, and to systematically evaluate how upstream continual learning can
rapidly generalize to new multimodal and unimodal tasks. CLiMB includes
implementations of several CL algorithms and a modified Vision-Language
Transformer (ViLT) model that can be deployed on both multimodal and unimodal
tasks. We find that common CL methods can help mitigate forgetting during
multimodal task learning, but do not enable cross-task knowledge transfer. We
envision that CLiMB will facilitate research on a new class of CL algorithms
for this challenging multimodal setting.

本研究介绍了 CLiMB 基准测试，并提出了一种改进的视觉 - 语言 Transformer (ViLT) 模型，用于同时处理多模态任务和单模态任务的连续学习问题。通过实验发现，虽然常见的连续学习方法可以缓解多模态任务学习中的遗忘问题，但并不能实现跨任务的知识转移。该基准测试将有助于研究这个多模态场景下的新型连续学习算法。