Pre-trained large-scale vision-language models (VLMs) have acquired profound
understanding of general visual concepts. Recent advancements in efficient
transfer learning (ETL) have shown remarkable success in fine-tuning VLMs
within the scenario of limited data, introducing only a few parameters to
harness task-specific insights from VLMs. Despite significant progress, current
leading ETL methods tend to overfit the narrow distributions of base classes
seen during training and encounter two primary challenges: (i) only utilizing
uni-modal information to modeling task-specific knowledge; and (ii) using
costly and time-consuming methods to supplement knowledge. To address these
issues, we propose a Conditional Prototype Rectification Prompt Learning (CPR)
method to correct the bias of base examples and augment limited data in an
effective way. Specifically, we alleviate overfitting on base classes from two
aspects. First, each input image acquires knowledge from both textual and
visual prototypes, and then generates sample-conditional text tokens. Second,
we extract utilizable knowledge from unlabeled data to further refine the
prototypes. These two strategies mitigate biases stemming from base classes,
yielding a more effective classifier. Extensive experiments on 11 benchmark
datasets show that our CPR achieves state-of-the-art performance on both
few-shot classification and base-to-new generalization tasks. Our code is
avaliable at https://github.com/chenhaoxing/CPR.

利用大规模的预先训练的视觉语言模型，通过高效的迁移学习方法解决基类过拟合问题，提出了一种条件原型纠正提示学习方法，有效地纠正了基类示例的偏差并增加了有限数据量，在少样本分类和从基类到新类的泛化任务中实现了最先进的性能。