Vision and Language Models (VLMs), such as CLIP, have enabled visual
recognition of a potentially unlimited set of categories described by text
prompts. However, for the best visual recognition performance, these models
still require tuning to better fit the data distributions of the downstream
tasks, in order to overcome the domain shift from the web-based pre-training
data. Recently, it has been shown that it is possible to effectively tune VLMs
without any paired data, and in particular to effectively improve VLMs visual
recognition performance using text-only training data generated by Large
Language Models (LLMs). In this paper, we dive deeper into this exciting
text-only VLM training approach and explore ways it can be significantly
further improved taking the specifics of the downstream task into account when
sampling text data from LLMs. In particular, compared to the SOTA text-only VLM
training approach, we demonstrate up to 8.4% performance improvement in (cross)
domain-specific adaptation, up to 8.7% improvement in fine-grained recognition,
and 3.1% overall average improvement in zero-shot classification compared to
strong baselines.

在此研究中，我们研究了一种基于文本的视觉与语言模型训练方法，并探索了如何根据下游任务的特点从大型语言模型中采样文本数据，以显著提升视觉识别性能。与先前方法相比，我们展示了达 (交) 领域特定适应的性能提高达 8.4％，精细化识别提高达 8.7％，零标记分类整体平均提高达 3.1％。

TAP：针对视觉分类任务自适应生成文本训练实例的有针对性提示

TAP: Targeted Prompting for Task Adaptive Generation of Textual Training  Instances for Visual Classification

We propose a novel approach for unsupervised zero-shot learning (ZSL) of
classes based on their names. Most existing unsupervised ZSL methods aim to
learn a model for directly comparing image features and class names. However,
this proves to be a difficult task due to dominance of non-visual semantics in
underlying vector-space embeddings of class names. To address this issue, we
discriminatively learn a word representation such that the similarities between
class and combination of attribute names fall in line with the visual
similarity. Contrary to the traditional zero-shot learning approaches that are
built upon attribute presence, our approach bypasses the laborious
attribute-class relation annotations for unseen classes. In addition, our
proposed approach renders text-only training possible, hence, the training can
be augmented without the need to collect additional image data. The
experimental results show that our method yields state-of-the-art results for
unsupervised ZSL in three benchmark datasets.

本论文提出了一种基于名称的无监督零样本学习方法，通过区别性地学习单词表示，使得类别和属性名称的相似性与视觉相似性相符，突破了传统零样本学习方法依赖属性特征标注的限制，并且能够在纯文本数据上进行训练，实验结果表明该方法在三个基准数据集上取得了最先进的性能。