Computer vision often treats perception as objective, and this assumption
gets reflected in the way that datasets are collected and models are trained.
For instance, image descriptions in different languages are typically assumed
to be translations of the same semantic content. However, work in
cross-cultural psychology and linguistics has shown that individuals differ in
their visual perception depending on their cultural background and the language
they speak. In this paper, we demonstrate significant differences in semantic
content across languages in both dataset and model-produced captions. When data
is multilingual as opposed to monolingual, captions have higher semantic
coverage on average, as measured by scene graph, embedding, and linguistic
complexity. For example, multilingual captions have on average 21.8% more
objects, 24.5% more relations, and 27.1% more attributes than a set of
monolingual captions. Moreover, models trained on content from different
languages perform best against test data from those languages, while those
trained on multilingual content perform consistently well across all evaluation
data compositions. Our research provides implications for how diverse modes of
perception can improve image understanding.

我们的研究展示了多语言语境下数据集和模型生成的图像标题之间的显著语义差异，以及不同语言训练的模型在对应语言的测试数据上表现最佳，而在多语言内容上训练的模型在所有评估数据组合上都表现良好，这对于改善图像理解的多样化感知具有重要意义。