With the utilization of Transformer architecture, large Vision and Language
(V&L) models have shown promising performance in even zero-shot settings.
Several studies, however, indicate a lack of robustness of the models when
dealing with complex linguistics and visual attributes. In this work, we
introduce a novel V&L benchmark - ColorFoil, by creating color-related foils to
assess the models' perception ability to detect colors like red, white, green,
etc. We evaluate seven state-of-the-art V&L models including CLIP, ViLT,
GroupViT, and BridgeTower, etc. in a zero-shot setting and present intriguing
findings from the V&L models. The experimental evaluation indicates that ViLT
and BridgeTower demonstrate much better color perception capabilities compared
to CLIP and its variants and GroupViT. Moreover, CLIP-based models and GroupViT
struggle to distinguish colors that are visually distinct to humans with normal
color perception ability.

利用 Transformer 架构，大型图像和语言（V&L）模型在零样本设置中展现出了令人期待的性能。然而，一些研究指出，在处理复杂的语言学和视觉属性时，这些模型的稳健性不足。在这项工作中，我们引入了一个新颖的 V&L 基准测试 ——ColorFoil，通过创建与颜色相关的误导以评估模型对于检测红色、白色、绿色等颜色的感知能力。我们在零样本设置中评估了包括 CLIP、ViLT、GroupViT 和 BridgeTower 等七种最先进的 V&L 模型，并从中得出了有趣的发现。实验评估表明，相比于 CLIP 及其变种和 GroupViT，ViLT 和 BridgeTower 表现出更好的颜色感知能力。而且，基于 CLIP 的模型和 GroupViT 难以区分在人类正常颜色感知能力下具有视觉差异的颜色。

ColorFoil：大规模视觉与语言模型中色盲调查

ColorFoil: Investigating Color Blindness in Large Vision and Language  Models

The need for grounding in language understanding is an active research topic.
Previous work has suggested that color perception and color language appear as
a suitable test bed to empirically study the problem, given its cognitive
significance and showing that there is considerable alignment between a defined
color space and the feature space defined by a language model. To further study
this issue, we collect a large scale source of colors and their descriptions,
containing almost a 1 million examples , and perform an empirical analysis to
compare two kinds of alignments: (i) inter-space, by learning a mapping between
embedding space and color space, and (ii) intra-space, by means of prompting
comparatives between color descriptions. Our results show that while color
space alignment holds for monolexemic, highly pragmatic color descriptions,
this alignment drops considerably in the presence of examples that exhibit
elements of real linguistic usage such as subjectivity and abstractedness,
suggesting that grounding may be required in such cases.

语言理解中的基础是一个活跃的研究领域。以颜色感知和颜色语言为研究基础，通过实证研究发现，颜色空间和语言模型定义的特征空间之间存在相当大的一致性。通过收集包含近 100 万个颜色及其描述的大规模数据集，进行两种对齐方式的实证分析：（i）跨空间对齐，即学习嵌入空间和颜色空间之间的映射，以及（ii）内部空间对齐，通过提示颜色描述之间的比较。我们的结果表明，尽管对于单词基础、高度实用的颜色描述，颜色空间对齐成立，但在存在主观性和抽象性等真实语言使用元素的示例中，这种对齐性显著降低，这表明在这种情况下可能需要基础。

无所依托的 LLMs 中的知觉结构：抽象性和主观性在颜色语言中的影响

Perceptual Structure in the Absence of Grounding for LLMs: The Impact of  Abstractedness and Subjectivity in Color Language

Deep neural networks (DNNs) are increasingly proposed as models of human
vision, bolstered by their impressive performance on image classification and
object recognition tasks. Yet, the extent to which DNNs capture fundamental
aspects of human vision such as color perception remains unclear. Here, we
develop novel experiments for evaluating the perceptual coherence of color
embeddings in DNNs, and we assess how well these algorithms predict human color
similarity judgments collected via an online survey. We find that
state-of-the-art DNN architectures $-$ including convolutional neural networks
and vision transformers $-$ provide color similarity judgments that strikingly
diverge from human color judgments of (i) images with controlled color
properties, (ii) images generated from online searches, and (iii) real-world
images from the canonical CIFAR-10 dataset. We compare DNN performance against
an interpretable and cognitively plausible model of color perception based on
wavelet decomposition, inspired by foundational theories in computational
neuroscience. While one deep learning model $-$ a convolutional DNN trained on
a style transfer task $-$ captures some aspects of human color perception, our
wavelet algorithm provides more coherent color embeddings that better predict
human color judgments compared to all DNNs we examine. These results hold when
altering the high-level visual task used to train similar DNN architectures
(e.g., image classification versus image segmentation), as well as when
examining the color embeddings of different layers in a given DNN architecture.
These findings break new ground in the effort to analyze the perceptual
representations of machine learning algorithms and to improve their ability to
serve as cognitively plausible models of human vision. Implications for machine
learning, human perception, and embodied cognition are discussed.

深度神经网络（DNN）在图像分类和物体识别任务中的表现令人印象深刻，然而它们对于颜色知觉等人类视觉基本方面的捕捉程度仍不清楚。本研究使用新型实验评估了 DNN 颜色嵌入的感知一致性，并通过在线调查评估了这些算法如何预测人类的颜色相似性判断。我们发现最先进的 DNN 架构，包括卷积神经网络和视觉变换器，所提供的颜色相似性判断与人类得出的（i）具有受控颜色属性的图像、（ii）从在线搜索生成的图像以及（iii）来自经典 CIFAR-10 数据集的现实世界图像存在显著差异。与我们检查的所有 DNN 相比，我们的基于小波分解的可解释和基于认知的颜色知觉模型提供更一致的颜色嵌入结果，更好地预测了人类的颜色判断。这些结果适用于在使用类似 DNN 体系结构进行训练的高级视觉任务（例如图像分类与图像分割）以及检查给定 DNN 体系结构中不同层的颜色嵌入时。这些发现在分析机器学习算法的感知表示和改进其作为人类视觉的认知合理模型的努力中创新了新的领域。机器学习、人类感知和具身认知的影响正在讨论中。