Transductive inference has been widely investigated in few-shot image
classification, but completely overlooked in the recent, fast growing
literature on adapting vision-langage models like CLIP. This paper addresses
the transductive zero-shot and few-shot CLIP classification challenge, in which
inference is performed jointly across a mini-batch of unlabeled query samples,
rather than treating each instance independently. We initially construct
informative vision-text probability features, leading to a classification
problem on the unit simplex set. Inspired by Expectation-Maximization (EM), our
optimization-based classification objective models the data probability
distribution for each class using a Dirichlet law. The minimization problem is
then tackled with a novel block Majorization-Minimization algorithm, which
simultaneously estimates the distribution parameters and class assignments.
Extensive numerical experiments on 11 datasets underscore the benefits and
efficacy of our batch inference approach.On zero-shot tasks with test batches
of 75 samples, our approach yields near 20% improvement in ImageNet accuracy
over CLIP's zero-shot performance. Additionally, we outperform state-of-the-art
methods in the few-shot setting. The code is available at:
this https URL

这篇论文研究了在 CLIP 等视觉语言模型的快速发展文献中被完全忽视的跨导式零样本和少样本 CLIP 分类挑战，其中推断是在一批未标记的查询样本上进行的，而不是独立处理每个实例。通过构建信息性的视觉文本概率特征，本文提出了一个基于最小化问题的分类优化目标，使用 Dirichlet 分布来建模每个类的数据概率分布。我们采用一种新颖的块主导次优化算法来解决这个最小化问题，该算法同时估计分布参数和类别分配。对 11 个数据集进行的大量数值实验证明了我们批处理推断方法的益处和功效。在测试批次包含 75 个样本的零样本任务中，我们的方法在 ImageNet 准确性方面相对于 CLIP 的零样本性能提高了近 20%。此外，在少样本设置中，我们超过了现有的最先进方法。代码可在此链接获得：this https URL