Class-Incremental Learning (CIL) or continual learning is a desired
capability in the real world, which requires a learning system to adapt to new
tasks without forgetting former ones. While traditional CIL methods focus on
visual information to grasp core features, recent advances in Vision-Language
Models (VLM) have shown promising capabilities in learning generalizable
representations with the aid of textual information. However, when continually
trained with new classes, VLMs often suffer from catastrophic forgetting of
former knowledge. Applying VLMs to CIL poses two major challenges: 1) how to
adapt the model without forgetting; and 2) how to make full use of the
multi-modal information. To this end, we propose PROjectiOn Fusion (PROOF) that
enables VLMs to learn without forgetting. To handle the first challenge, we
propose training task-specific projections based on the frozen image/text
encoders. When facing new tasks, new projections are expanded and former
projections are fixed, alleviating the forgetting of old concepts. For the
second challenge, we propose the fusion module to better utilize the
cross-modality information. By jointly adjusting visual and textual features,
the model can capture semantic information with stronger representation
ability. Extensive experiments on nine benchmark datasets validate PROOF
achieves state-of-the-art performance.

我们提出了一种叫做 PROOF 的模型，它通过训练任务特定的映射来解决 Vision-Language Models 在 Class-Incremental Learning 时候遗忘问题，并且通过融合多模态信息来提高模型的语义表示能力。在九个基准数据集上进行的实验表明，PROOF 达到了最先进的性能。