Fine-tuning vision-language models (VLMs) with abundant unlabeled data
recently has attracted increasing attention. Existing methods that resort to
the pseudolabeling strategy would suffer from heavily incorrect hard
pseudolabels when VLMs exhibit low zero-shot performance in downstream tasks.
To alleviate this issue, we propose a Candidate Pseudolabel Learning method,
termed CPL, to fine-tune VLMs with suitable candidate pseudolabels of unlabeled
data in downstream tasks. The core of our method lies in the generation
strategy of candidate pseudolabels, which progressively generates refined
candidate pseudolabels by both intra- and inter-instance label selection, based
on a confidence score matrix for all unlabeled data. This strategy can result
in better performance in true label inclusion and class-balanced instance
selection. In this way, we can directly apply existing loss functions to learn
with generated candidate psueudolabels. Extensive experiments on nine benchmark
datasets with three learning paradigms demonstrate the effectiveness of our
method. Our code can be found at this https URL

通过候选伪标签学习方法（CPL）在下游任务中使用适当的候选伪标签细调视觉 - 语言模型（VLMs），以提高 VLMs 在大量无标签数据上的 True 标签包含能力和类别平衡实例选择效果。