The emergence of vision-language models (VLMs), such as CLIP, has spurred a
significant research effort towards their application for downstream supervised
learning tasks. Although some previous studies have explored the unsupervised
fine-tuning of CLIP, they often rely on prior knowledge in the form of class
names associated with ground truth labels. In this paper, we delve into a
realistic unsupervised fine-tuning scenario by assuming that the unlabeled data
might contain out-of-distribution samples from unknown classes. Furthermore, we
emphasize the importance of simultaneously enhancing out-of-distribution
detection capabilities alongside the recognition of instances associated with
predefined class labels.
To tackle this problem, we present a simple, efficient, and effective
fine-tuning approach called Universal Entropy Optimization (UEO). UEO leverages
sample-level confidence to approximately minimize the conditional entropy of
confident instances and maximize the marginal entropy of less confident
instances. Apart from optimizing the textual prompts, UEO also incorporates
optimization of channel-wise affine transformations within the visual branch of
CLIP. Through extensive experiments conducted across 15 domains and 4 different
types of prior knowledge, we demonstrate that UEO surpasses baseline methods in
terms of both generalization and out-of-distribution detection.

通过将视觉语言模型 (VLMs) 应用于下游监督学习任务，本文探讨了无监督微调 CLIP 模型，解决了未知类别的样本和识别预定义类别实例的问题，并提出了一种称为通用熵优化 (UEO) 的简单有效的微调方法。通过广泛的实验，我们证明了 UEO 方法在泛化能力和检测未知类别样本方面优于基线方法。