Current state-of-the-art vision-and-language models are evaluated on tasks
either individually or in a multi-task setting, overlooking the challenges of
continually learning (CL) tasks as they arrive. Existing CL benchmarks have
facilitated research on task adaptation and mitigating "catastrophic
forgetting", but are limited to vision-only and language-only tasks. We present
CLiMB, a benchmark to study the challenge of learning multimodal tasks in a CL
setting, and to systematically evaluate how upstream continual learning can
rapidly generalize to new multimodal and unimodal tasks. CLiMB includes
implementations of several CL algorithms and a modified Vision-Language
Transformer (ViLT) model that can be deployed on both multimodal and unimodal
tasks. We find that common CL methods can help mitigate forgetting during
multimodal task learning, but do not enable cross-task knowledge transfer. We
envision that CLiMB will facilitate research on a new class of CL algorithms
for this challenging multimodal setting.

本研究介绍了 CLiMB 基准测试，并提出了一种改进的视觉 - 语言 Transformer (ViLT) 模型，用于同时处理多模态任务和单模态任务的连续学习问题。通过实验发现，虽然常见的连续学习方法可以缓解多模态任务学习中的遗忘问题，但并不能实现跨任务的知识转移。该基准测试将有助于研究这个多模态场景下的新型连续学习算法。

CLiMB: 用于视觉语言任务的持续学习基准

CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks

Most recently proposed methods for Neural Program Induction work under the
assumption of having a large set of input/output (I/O) examples for learning
any underlying input-output mapping. This paper aims to address the problem of
data and computation efficiency of program induction by leveraging information
from related tasks. Specifically, we propose two approaches for cross-task
knowledge transfer to improve program induction in limited-data scenarios. In
our first proposal, portfolio adaptation, a set of induction models is
pretrained on a set of related tasks, and the best model is adapted towards the
new task using transfer learning. In our second approach, meta program
induction, a $k$-shot learning approach is used to make a model generalize to
new tasks without additional training. To test the efficacy of our methods, we
constructed a new benchmark of programs written in the Karel programming
language. Using an extensive experimental evaluation on the Karel benchmark, we
demonstrate that our proposals dramatically outperform the baseline induction
method that does not use knowledge transfer. We also analyze the relative
performance of the two approaches and study conditions in which they perform
best. In particular, meta induction outperforms all existing approaches under
extreme data sparsity (when a very small number of examples are available),
i.e., fewer than ten. As the number of available I/O examples increase (i.e. a
thousand or more), portfolio adapted program induction becomes the best
approach. For intermediate data sizes, we demonstrate that the combined method
of adapted meta program induction has the strongest performance.

该研究旨在通过跨任务知识转移的两种方法 —— 投资组合适应和元编程归纳，以提高节约数据的编程归纳的效率，并针对一个新的基准程序进行广泛的实验评估。实验评估结果表明，这两种方法明显优于那些不使用知识转移的编程归纳基准方法，并分析了它们的表现最佳的条件。