Prompt learning has become the most effective paradigm for adapting large
pre-trained vision-language models (VLMs) to downstream tasks. Recently,
unsupervised prompt tuning methods, such as UPL and POUF, directly leverage
pseudo-labels as supervisory information to fine-tune additional adaptation
modules on unlabeled data. However, inaccurate pseudo labels easily misguide
the tuning process and result in poor representation capabilities. In light of
this, we propose Training-Free Unsupervised Prompts (TFUP), which maximally
preserves the inherent representation capabilities and enhances them with a
residual connection to similarity-based prediction probabilities in a
training-free and labeling-free manner. Specifically, we integrate both
instance confidence and prototype scores to select representative samples,
which are used to customize a reliable Feature Cache Model (FCM) for
training-free inference. Then, we design a Multi-level Similarity Measure (MSM)
that considers both feature-level and semantic-level similarities to calculate
the distance between each test image and the cached sample as the weight of the
corresponding cached label to generate similarity-based prediction
probabilities. In this way, TFUP achieves surprising performance, even
surpassing the training-base method on multiple classification datasets. Based
on our TFUP, we propose a training-based approach (TFUP-T) to further boost the
adaptation performance. In addition to the standard cross-entropy loss, TFUP-T
adopts an additional marginal distribution entropy loss to constrain the model
from a global perspective. Our TFUP-T achieves new state-of-the-art
classification performance compared to unsupervised and few-shot adaptation
approaches on multiple benchmarks. In particular, TFUP-T improves the
classification accuracy of POUF by 3.3% on the most challenging Domain-Net
dataset.

从大量的预先训练图像 - 语言模型 (VLMs) 调整到下游任务的适应性最好的范式变成了即刻学习。我们提出了无需训练的无监督提示 (TFUP)，它在无需训练或标记的情况下最大程度地保留了内在的表现能力，并通过将预测概率与基于相似度的预测概率之间的残差连接进行增强。然后，我们使用实例置信度和原型分数来选择代表性样本，这些样本用于定制训练自由推理的可靠特征缓存模型 (FCM)。我们设计了一种多级相似度度量 (MSM)，它考虑了特征级和语义级相似度，以计算每个测试图像与缓存样本之间的距离，作为相应缓存标签的权重，进而生成基于相似度的预测概率。通过这种方式，TFUP 在多个分类数据集上实现了令人惊讶的性能，甚至超过了基于训练的方法。在我们的 TFUP 基础上，我们提出了一个训练为基础的方法 (TFUP-T) 来进一步提升适应性能力。除了标准的交叉熵损失外，TFUP-T 还采用了额外的边际分布熵损失，从全局角度约束模型。与无监督和少样本适应方法在多个基准测试中相比，我们的 TFUP-T 实现了新的最先进的分类性能。特别是，在最具挑战的 Domain-Net 数据集上，TFUP-T 将 POUF 的分类准确度提高了 3.3%。