This paper explores the grounding issue concerning multimodal semantic
representation from a computational cognitive-linguistic view. Five perceptual
properties of groundedness are annotated and analyzed: Affordance, Perceptual
salience, Object number, Gaze cueing, and Ecological Niche Association (ENA).
We annotated selected images from the Flickr30k dataset with exploratory
analyses and statistical modeling of their captions. Our findings suggest that
a comprehensive understanding of an object or event requires cognitive
attention, semantic distinctions in linguistic expression, and multimodal
construction. During this construction process, viewers integrate situated
meaning and affordance into multimodal semantics, which is consolidated into
image captions used in the image-text dataset incorporating visual and textual
elements. Our findings suggest that situated meaning and affordance grounding
are critical for grounded natural language understanding systems to generate
appropriate responses and show the potential to advance the understanding of
human construal in diverse situations.

本文从计算认知语言学的角度探讨了多模态语义表示中的基础问题，注释和分析了五个感知属性：可操作性、感知显著性、物体数量、凝视引导和生态学生态位关联。我们对来自 Flickr30k 数据集的选定图像进行了探究性分析和统计建模，发现了在构建过程中，观看者将情境意义和可操作性与多模态语义统合到一起，而这些因素被巩固为包含视觉和文本元素的图像文字数据集中的影像字幕。本研究显示情境意义和可操作性基础对于理解系统生成适当回应至关重要，并有潜力推动人类对不同情境下的构造的理解。

探索图像字幕中的基础问题

Exploring the Grounding Issues in Image Caption

Neural language models encode rich knowledge about entities and their
relationships which can be extracted from their representations using probing.
Common properties of nouns (e.g., red strawberries, small ant) are, however,
more challenging to extract compared to other types of knowledge because they
are rarely explicitly stated in texts. We hypothesize this to mainly be the
case for perceptual properties which are obvious to the participants in the
communication. We propose to extract these properties from images and use them
in an ensemble model, in order to complement the information that is extracted
from language models. We consider perceptual properties to be more concrete
than abstract properties (e.g., interesting, flawless). We propose to use the
adjectives' concreteness score as a lever to calibrate the contribution of each
source (text vs. images). We evaluate our ensemble model in a ranking task
where the actual properties of a noun need to be ranked higher than other
non-relevant properties. Our results show that the proposed combination of text
and images greatly improves noun property prediction compared to powerful
text-based language models.

本文提出了一种利用图像来补充语言模型信息的方法，并在排名任务中验证了该模型在名词属性预测上的优越性，相较于仅使用文本的语言模型。

基于具体性的合集模型用于名词属性预测的可视化

Visualizing the Obvious: A Concreteness-based Ensemble Model for Noun Property Prediction

Deep generative models can emulate the perceptual properties of complex image
datasets, providing a latent representation of the data. However, manipulating
such representation to perform meaningful and controllable transformations in
the data space remains challenging without some form of supervision. While
previous work has focused on exploiting statistical independence to disentangle
latent factors, we argue that such requirement is too restrictive and propose
instead a non-statistical framework that relies on counterfactual manipulations
to uncover a modular structure of the network composed of disentangled groups
of internal variables. Experiments with a variety of generative models trained
on complex image datasets show the obtained modules can be used to design
targeted interventions. This opens the way to applications such as
computationally efficient style transfer and the automated assessment of
robustness to contextual changes in pattern recognition systems.

本研究提出了一种非统计框架，它使用反事实干预来揭示由解缠组成的网络的模块化结构，以便操纵潜在表示以执行有意义和可控的变换。通过在复杂图像数据集上训练各种生成模型进行实验，表明所获得的模块可用于设计目标干预，从而在风格转换和自动评估对环境变化的稳健性等领域开展应用。