A promising approach for improving the performance of vision-language models like CLIP for image classification is to extend the class descriptions (i.e., prompts) with related attributes, e.g., using brown sparrow instead of sparrow. However, current zero-shot methods select a subset of attributes regardless of commonalities between the target classes, potentially providing no useful information that would have helped to distinguish between them. For instance, they may use color instead of bill shape to distinguish between sparrows and wrens, which are both brown. We propose Follow-up Differential Descriptions (FuDD), a zero-shot approach that tailors the class descriptions to each dataset and leads to additional attributes that better differentiate the target classes. FuDD first identifies the ambiguous classes for each image, and then uses a Large Language Model (LLM) to generate new class descriptions that differentiate between them. The new class descriptions resolve the initial ambiguity and help predict the correct label. In our experiments, FuDD consistently outperforms generic description ensembles and naive LLM-generated descriptions on 12 datasets. We show that differential descriptions are an effective tool to resolve class ambiguities, which otherwise significantly degrade the performance. We also show that high quality natural language class descriptions produced by FuDD result in comparable performance to few-shot adaptation methods.

一种提高图像分类的视觉语言模型（如CLIP）性能的有希望方法是通过扩展类别描述（即提示）与相关属性结合，通过使用棕色麻雀代替麻雀等方式。然而，当前的零样本方法无论目标类别之间的共同点如何，仍然会选择一组属性，可能提供了无法区分它们的有用信息。我们提出了一种名为Follow-up Differential Descriptions（FuDD）的零样本方法，该方法为每个数据集量身定制类别描述，并生成更好区分目标类别的新属性。在实验中，FuDD在12个数据集上始终优于通用描述集合和朴素的LLM生成描述，表明不同描述是解决类别模糊的有效工具，否则会显著降低性能。我们还展示了FuDD生成的高质量自然语言类别描述与少样本适应方法在性能上有可比性。

跟进性差异描述：语言模型消除图像分类中的歧义