Transformer neural networks are increasingly replacing prior architectures in
a wide range of applications in different data modalities. The increasing size
and computational demands of fine-tuning large pre-trained transformer neural
networks pose significant challenges for the widespread adoption of these
models for applications that demand on-edge computing. To tackle this
challenge, continual learning (CL) emerges as a solution by facilitating the
transfer of knowledge across tasks that arrive sequentially for an autonomously
learning agent. However, current CL methods mainly focus on learning tasks that
are exclusively vision-based or language-based. We propose a transformer-based
CL framework focusing on learning tasks that involve both vision and language,
known as Vision-and-Language (VaL) tasks. Due to the success of transformers in
other modalities, our architecture has the potential to be used in multimodal
learning settings. In our framework, we benefit from introducing extra
parameters to a base transformer to specialize the network for each task. As a
result, we enable dynamic model expansion to learn several tasks in a sequence.
We also use knowledge distillation to benefit from relevant past experiences to
learn the current task more efficiently. Our proposed method, Task Attentive
Multimodal Continual Learning (TAM-CL), allows for the exchange of information
between tasks while mitigating the problem of catastrophic forgetting. Notably,
our approach is scalable, incurring minimal memory and time overhead. TAM-CL
achieves state-of-the-art (SOTA) performance on challenging multimodal tasks

我们提出了一种基于 Transformer 的持续学习框架 TAM-CL，用于学习涉及视觉和语言的多模态任务，并通过引入额外参数和知识蒸馏实现任务间的信息交流，以及解决灾难性遗忘问题。该方法在多种挑战性的多模态任务上达到了最先进的性能。