Recently, large-scale pre-trained Vision and Language (VL) models have set a
new state-of-the-art (SOTA) in zero-shot visual classification enabling
open-vocabulary recognition of potentially unlimited set of categories defined
as simple language prompts. However, despite these great advances, the
performance of these zeroshot classifiers still falls short of the results of
dedicated (closed category set) classifiers trained with supervised fine
tuning. In this paper we show, for the first time, how to reduce this gap
without any labels and without any paired VL data, using an unlabeled image
collection and a set of texts auto-generated using a Large Language Model (LLM)
describing the categories of interest and effectively substituting labeled
visual instances of those categories. Using our label-free approach, we are
able to attain significant performance improvements over the zero-shot
performance of the base VL model and other contemporary methods and baselines
on a wide variety of datasets, demonstrating absolute improvement of up to
11.7% (3.8% on average) in the label-free setting. Moreover, despite our
approach being label-free, we observe 1.3% average gains over leading few-shot
prompting baselines that do use 5-shot supervision.

本文介绍了一种使用未标记的图像集合和大型语言模型自动生成标签，并通过这种非监督方式实现了零样本分类器性能的显著提高的方法。与传统的监督训练方法相比，在多个数据集上的绝对提升高达 11.7％（平均 3.8％），而与一些少样本提示基线相比的平均增益为 1.3％。

LaFTer：使用语言和未标记图像集合进行零样本分类器的无标签调整

LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and  Unlabeled Image Collections

We consider the problem of discovering novel object categories in an image
collection. While these images are unlabelled, we also assume prior knowledge
of related but different image classes. We use such prior knowledge to reduce
the ambiguity of clustering, and improve the quality of the newly discovered
classes. Our contributions are twofold. The first contribution is to extend
Deep Embedded Clustering to a transfer learning setting; we also improve the
algorithm by introducing a representation bottleneck, temporal ensembling, and
consistency. The second contribution is a method to estimate the number of
classes in the unlabelled data. This also transfers knowledge from the known
classes, using them as probes to diagnose different choices for the number of
classes in the unlabelled subset. We thoroughly evaluate our method,
substantially outperforming state-of-the-art techniques in a large number of
benchmarks, including ImageNet, OmniGlot, CIFAR-100, CIFAR-10, and SVHN.

该研究考虑了在图像集合中发现新对象类别的问题，使用先前的相关但不同的图像类别的先验知识降低聚类的歧义，改善新发现类别的质量，其中使用了 Deep Embedded Clustering、传输学习、表示瓶颈、时间集成、一致性和类别估计等多种技巧，并在 ImageNet、OmniGlot、CIFAR-100、CIFAR-10 和 SVHN 等许多基准测试中显著优于现有技术。