Recently, large-scale pre-trained Vision and Language (VL) models have set a
new state-of-the-art (SOTA) in zero-shot visual classification enabling
open-vocabulary recognition of potentially unlimited set of categories defined
as simple language prompts. However, despite these great advances, the
performance of these zeroshot classifiers still falls short of the results of
dedicated (closed category set) classifiers trained with supervised fine
tuning. In this paper we show, for the first time, how to reduce this gap
without any labels and without any paired VL data, using an unlabeled image
collection and a set of texts auto-generated using a Large Language Model (LLM)
describing the categories of interest and effectively substituting labeled
visual instances of those categories. Using our label-free approach, we are
able to attain significant performance improvements over the zero-shot
performance of the base VL model and other contemporary methods and baselines
on a wide variety of datasets, demonstrating absolute improvement of up to
11.7% (3.8% on average) in the label-free setting. Moreover, despite our
approach being label-free, we observe 1.3% average gains over leading few-shot
prompting baselines that do use 5-shot supervision.

本文介绍了一种使用未标记的图像集合和大型语言模型自动生成标签，并通过这种非监督方式实现了零样本分类器性能的显著提高的方法。与传统的监督训练方法相比，在多个数据集上的绝对提升高达 11.7％（平均 3.8％），而与一些少样本提示基线相比的平均增益为 1.3％。

LaFTer：使用语言和未标记图像集合进行零样本分类器的无标签调整

LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and  Unlabeled Image Collections

Histological analysis of tissue samples is one of the most widely used
methods for disease diagnosis. After taking a sample from a patient, it goes
through a lengthy and laborious preparation, which stains the tissue to
visualize different histological features under a microscope. Here, we
demonstrate a label-free approach to create a virtually-stained microscopic
image using a single wide-field auto-fluorescence image of an unlabeled tissue
sample, bypassing the standard histochemical staining process, saving time and
cost. This method is based on deep learning, and uses a convolutional neural
network trained using a generative adversarial network model to transform an
auto-fluorescence image of an unlabeled tissue section into an image that is
equivalent to the bright-field image of the stained-version of the same sample.
We validated this method by successfully creating virtually-stained microscopic
images of human tissue samples, including sections of salivary gland, thyroid,
kidney, liver and lung tissue, also covering three different stains. This
label-free virtual-staining method eliminates cumbersome and costly
histochemical staining procedures, and would significantly simplify tissue
preparation in pathology and histology fields.

本研究使用基于深度学习的生成对抗网络模型，将未标记组织样本的自动荧光图像转化为等效于其染色样本的亮场图像，实现了一种免标记虚拟染色的显微镜图像制作方法，有效节约了组织学检查的时间和成本。