Humans show language-biased image recognition for a word-embedded image,
known as picture-word interference. Such interference depends on hierarchical
semantic categories and reflects that human language processing highly
interacts with visual processing. Similar to humans, recent artificial models
jointly trained on texts and images, e.g., OpenAI CLIP, show language-biased
image classification. Exploring whether the bias leads to interference similar
to those observed in humans can contribute to understanding how much the model
acquires hierarchical semantic representations from joint learning of language
and vision. The present study introduces methodological tools from the
cognitive science literature to assess the biases of artificial models.
Specifically, we introduce a benchmark task to test whether words superimposed
on images can distort the image classification across different category levels
and, if it can, whether the perturbation is due to the shared semantic
representation between language and vision. Our dataset is a set of
word-embedded images and consists of a mixture of natural image datasets and
hierarchical word labels with superordinate/basic category levels. Using this
benchmark test, we evaluate the CLIP model. We show that presenting words
distorts the image classification by the model across different category
levels, but the effect does not depend on the semantic relationship between
images and embedded words. This suggests that the semantic word representation
in the CLIP visual processing is not shared with the image representation,
although the word representation strongly dominates for word-embedded images.

通过引入基于认知科学文献的方法工具，本研究介绍了一项基准测试来评估人工模型的偏差，并使用这个基准测试评估了 CLIP 模型。我们发现，虽然单词嵌入的图像会扭曲 CLIP 模型在不同类别级别上的图像分类，但这种影响不依赖于图像和嵌入单词之间的语义关系，这表明 CLIP 视觉处理中的语义词表示与图像表示不共享。