Although the mapping between sound and meaning in human language is assumed
to be largely arbitrary, research in cognitive science has shown that there are
non-trivial correlations between particular sounds and meanings across
languages and demographic groups, a phenomenon known as sound symbolism. Among
the many dimensions of meaning, sound symbolism is particularly salient and
well-demonstrated with regards to cross-modal associations between language and
the visual domain. In this work, we address the question of whether sound
symbolism is reflected in vision-and-language models such as CLIP and Stable
Diffusion. Using zero-shot knowledge probing to investigate the inherent
knowledge of these models, we find strong evidence that they do show this
pattern, paralleling the well-known kiki-bouba effect in psycholinguistics. Our
work provides a novel method for demonstrating sound symbolism and
understanding its nature using computational tools. Our code will be made
publicly available.

使用计算工具，我们研究了声音象征的存在并且证明了视觉与语言之间存在交叉模态关联，其表现与著名的 kiki-bouba 效应相似。

视听语言模型中的声音象征：Kiki 还是 Bouba？

Kiki or Bouba? Sound Symbolism in Vision-and-Language Models

This study explores an interesting pattern emerging from research that
combines artificial intelligence with sound symbolism. In these studies,
supervised machine learning algorithms are trained to classify samples based on
the sounds of referent names. Machine learning algorithms are efficient
learners of sound symbolism, but they tend to bias one category over the other.
The pattern is this: when a category arguably represents greater threat, the
algorithms tend to overpredict to that category. A hypothesis, framed by error
management theory, is presented that proposes that this may be evidence of an
adaptation to preference cautious behaviour. This hypothesis is tested by
constructing extreme gradient boosted (XGBoost) models using the sounds that
make up the names of Chinese, Japanese and Korean Pokemon and observing
classification error distribution.

研究人工智能和声音象征主义之间的结合，以及机器学习算法在训练中的偏向性和错误管理理论的假设，在中日韩宝可梦的名称中观察分类误差分布。