Continual learning focuses on incrementally training a model on a sequence of
tasks with the aim of learning new tasks while minimizing performance drop on
previous tasks. Existing approaches at the intersection of Continual Learning
and Visual Question Answering (VQA) do not study how the multimodal nature of
the input affects the learning dynamics of a model. In this paper, we
demonstrate that each modality evolves at different rates across a continuum of
tasks and that this behavior occurs in established encoder-only models as well
as modern recipes for developing Vision & Language (VL) models. Motivated by
this observation, we propose a modality-aware feature distillation (MAFED)
approach which outperforms existing baselines across models of varying scale in
three multimodal continual learning settings. Furthermore, we provide ablations
showcasing that modality-aware distillation complements experience replay.
Overall, our results emphasize the importance of addressing modality-specific
dynamics to prevent forgetting in multimodal continual learning.

模态感知特征蒸馏在多模态连续学习中表现出优于现有基准方法的结果，并强调了解决模态特定动态性的重要性以防止遗忘。