With the advent of large-scale pre-trained models, interest in adapting and
exploiting them for continual learning scenarios has grown.
In this paper, we propose an approach to exploiting pre-trained
vision-language models (e.g. CLIP) that enables further adaptation instead of
only using zero-shot learning of new tasks. We augment a pre-trained CLIP model
with additional layers after the Image Encoder or before the Text Encoder. We
investigate three different strategies: a Linear Adapter, a Self-attention
Adapter, each operating on the image embedding, and Prompt Tuning which instead
modifies prompts input to the CLIP text encoder. We also propose a method for
parameter retention in the adapter layers that uses a measure of parameter
importance to better maintain stability and plasticity during incremental
learning. Our experiments demonstrate that the simplest solution -- a single
Linear Adapter layer with parameter retention -- produces the best results.
Experiments on several conventional benchmarks consistently show a significant
margin of improvement over the current state-of-the-art.

通过在图像编码器之后或文本编码器之前在预训练的 CLIP 模型上增加附加层，我们提出了一种利用预训练视觉 - 语言模型进行进一步调整的方法，从而使其能够适应新任务而不仅仅是零样本学习。我们对线性适配器、自注意适配器以及修改 CLIP 文本编码器输入的提示调整进行了研究。此外，我们还提出了一种参数保留的方法，通过衡量参数重要性，在增量学习过程中更好地保持稳定性和可塑性。实验证明，最简单的解决方案 —— 一个具有参数保留的线性适配器层，获得了最佳结果。多个常规基准实验一致表明这种方法显著改进了现有技术水平。