Large vision-language models (VLMs), such as CLIP, have become foundational,
demonstrating remarkable success across a variety of downstream tasks. Despite
their advantages, these models, akin to other foundational systems, inherit
biases from the disproportionate distribution of real-world data, leading to
misconceptions about the actual environment. Prevalent datasets like ImageNet
are often riddled with non-causal, spurious correlations that can diminish VLM
performance in scenarios where these contextual elements are absent. This study
presents an investigation into how a simple linear probe can effectively
distill task-specific core features from CLIP's embedding for downstream
applications. Our analysis reveals that the CLIP text representations are often
tainted by spurious correlations, inherited in the biased pre-training dataset.
Empirical evidence suggests that relying on visual representations from CLIP,
as opposed to text embedding, is more practical to refine the skewed
perceptions in VLMs, emphasizing the superior utility of visual representations
in overcoming embedded biases. Our codes will be available here.

通过使用简单的线性探测器，本研究探讨了如何从 CLIP 的嵌入中有效地提取特定任务的核心特征，结果显示 CLIP 的文本表示往往受到了有偏向的预训练数据中具有误导性的相关性的影响，实证结果表明相比文本嵌入，依赖于 CLIP 的视觉表示更为实用，可克服内置偏见。

通过视觉表示精炼视觉 - 语言模型中的偏差感知

Refining Skewed Perceptions in Vision-Language Models through Visual  Representations

In a recent, strongly emergent literature on few-shot CLIP adaptation, Linear
Probe (LP) has been often reported as a weak baseline. This has motivated
intensive research building convoluted prompt learning or feature adaptation
strategies. In this work, we propose and examine from convex-optimization
perspectives a generalization of the standard LP baseline, in which the linear
classifier weights are learnable functions of the text embedding, with
class-wise multipliers blending image and text knowledge. As our objective
function depends on two types of variables, i.e., the class visual prototypes
and the learnable blending parameters, we propose a computationally efficient
block coordinate Majorize-Minimize (MM) descent algorithm. In our full-batch MM
optimizer, which we coin LP++, step sizes are implicit, unlike standard
gradient descent practices where learning rates are intensively searched over
validation sets. By examining the mathematical properties of our loss (e.g.,
Lipschitz gradient continuity), we build majorizing functions yielding
data-driven learning rates and derive approximations of the loss's minima,
which provide data-informed initialization of the variables. Our image-language
objective function, along with these non-trivial optimization insights and
ingredients, yields, surprisingly, highly competitive few-shot CLIP
performances. Furthermore, LP++ operates in black-box, relaxes intensive
validation searches for the optimization hyper-parameters, and runs
orders-of-magnitudes faster than state-of-the-art few-shot CLIP adaptation
methods. Our code is available at:
https://github.com/FereshteShakeri/FewShot-CLIP-Strong-Baseline.git.

在这项研究中，我们提出并从凸优化的角度考察了标准线性探测法（LP）基线的一个泛化模型，其中线性分类器权重是文本嵌入的可学习函数，借助类别倍增器混合图像和文本知识。通过引入块协调主罩 - 最小化下降算法，我们提出一个计算高效的全批次优化器 LP++，其步长是隐式的，不像标准梯度下降方法那样需要在验证集上密集搜索学习率。通过研究我们的损失函数的数学特性（例如 Lipschitz 梯度连续性），我们构建了一些主罩函数，得到了数据驱动的学习率，并得到损失最小值的近似值，从而为变量提供了数据驱动的初始化。我们的图像 - 语言目标函数以及这些非平凡的优化洞察和要素出人意料地提供了高度竞争的少样本 CLIP 性能。此外，LP++ 以黑盒的方式运行，放宽了对优化超参数的密集验证搜索，并且比最先进的少样本 CLIP 适应方法运行速度快几个数量级。我们的代码可在以下网址找到：https://github.com/FereshteShakeri/FewShot-CLIP-Strong-Baseline.git。

LP++：一种令人惊讶的强大线性探针用于 Few-Shot CLIP

LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIP

This work explores whether language models encode meaningfully grounded
representations of sounds of objects. We learn a linear probe that retrieves
the correct text representation of an object given a snippet of audio related
to that object, where the sound representation is given by a pretrained audio
model. This probe is trained via a contrastive loss that pushes the language
representations and sound representations of an object to be close to one
another. After training, the probe is tested on its ability to generalize to
objects that were not seen during training. Across different language models
and audio models, we find that the probe generalization is above chance in many
cases, indicating that despite being trained only on raw text, language models
encode grounded knowledge of sounds for some objects.

这项研究探讨了语言模型是否能够在对象的声音上编码具有意义的基于意思的表示。通过学习一个线性探针，该探针可以在给定与对象相关的音频片段时检索到对象的正确文本表示，其中声音表示由预训练的音频模型提供。这个探针通过对比损失进行训练，使得对象的语言表示和声音表示彼此靠近。在训练后，我们测试了该探针在推广到训练过程中未见过的对象上的能力。通过不同的语言模型和音频模型，我们发现在许多情况下，探针的推广能力高于随机情况，这表明尽管仅在原始文本上进行训练，语言模型为某些对象的声音编码了有根据的知识。