With the growing interest in pretrained vision-language models like CLIP,
recent research has focused on adapting these models to downstream tasks.
Despite achieving promising results, most existing methods require labeled data
for all classes, which may not hold in real-world applications due to the long
tail and Zipf's law. For example, some classes may lack labeled data entirely,
such as emerging concepts. To address this problem, we propose a plug-and-play
generative approach called \textbf{S}ynt\textbf{H}es\textbf{I}zed
\textbf{P}rompts~(\textbf{SHIP}) to improve existing fine-tuning methods.
Specifically, we follow variational autoencoders to introduce a generator that
reconstructs the visual features by inputting the synthesized prompts and the
corresponding class names to the textual encoder of CLIP. In this manner, we
easily obtain the synthesized features for the remaining label-only classes.
Thereafter, we fine-tune CLIP with off-the-shelf methods by combining labeled
and synthesized features. Extensive experiments on base-to-new generalization,
cross-dataset transfer learning, and generalized zero-shot learning demonstrate
the superiority of our approach. The code is available at
https://github.com/mrflogs/SHIP.

本文提出了一种基于生成式方法的模型适配方案 (SHIP)，使用文本和图像信息进行训练的预训练模型 (CLIP) 可以在没有标签的类别上表现出更好的效果。在对基础数据集到新的数据集的泛化、跨数据集的迁移学习和广义的零样本学习等方面进行了广泛实验，证明了该方法的优越性