Contrastive pretrained large Vision-Language Models (VLMs) like CLIP have
revolutionized visual representation learning by providing good performance on
downstream datasets. VLMs are 0-shot adapted to a downstream dataset by
designing prompts that are relevant to the dataset. Such prompt engineering
makes use of domain expertise and a validation dataset. Meanwhile, recent
developments in generative pretrained models like GPT-4 mean they can be used
as advanced internet search tools. They can also be manipulated to provide
visual information in any structure. In this work, we show that GPT-4 can be
used to generate text that is visually descriptive and how this can be used to
adapt CLIP to downstream tasks. We show considerable improvements in 0-shot
transfer accuracy on specialized fine-grained datasets like EuroSAT (~7%), DTD
(~7%), SUN397 (~4.6%), and CUB (~3.3%) when compared to CLIP's default prompt.
We also design a simple few-shot adapter that learns to choose the best
possible sentences to construct generalizable classifiers that outperform the
recently proposed CoCoOP by ~2% on average and by over 4% on 4 specialized
fine-grained datasets. We will release the code, prompts, and auxiliary text
dataset upon acceptance.

我们展示了如何使用 GPT-4 生成视觉描述性文本，并说明如何将其用于适应 CLIP 进行下游任务。与 CLIP 的默认提示相比，在专门的细粒度数据集上，我们在 0-shot 传输准确性方面取得了显著的改进。我们还设计了一个简单的 few-shot 适配器，学习选择最佳的句子来构建具有优越性能的可推广分类器。