We consider the problem of adapting a contrastively pretrained vision-language model like CLIP (Radford et al., 2021) for few-shot classification. The existing literature addresses this problem by learning a linear classifier of the frozen visual features, optimizing word embeddings, or learning external feature adapters. This paper introduces an alternative way for CLIP adaptation without adding 'external' parameters to optimize. We find that simply fine-tuning the last projection matrix of the vision encoder leads to strong performance compared to the existing baselines. Furthermore, we show that regularizing training with the distance between the fine-tuned and pretrained matrices adds reliability for adapting CLIP through this layer. Perhaps surprisingly, this approach, coined ProLIP, yields performances on par or better than state of the art on 11 few-shot classification benchmarks, few-shot domain generalization, cross-dataset transfer and test-time adaptation. Code will be made available at https://github.com/astra-vision/ProLIP .

本研究解决了如何将对比预训练的视觉语言模型CLIP适应于少样本分类的问题。我们提出了一种新的方法，通过微调视觉编码器的最后投影矩阵，而不引入额外的优化参数，从而在多个基准测试中获得与现有最优方案相当或更优的性能。这一方法可能会推动少样本分类和领域泛化的研究前沿。

微调CLIP的最后视觉投影器：少样本的丰富性