Currently, dialogue systems have achieved high performance in processing
text-based communication. However, they have not yet effectively incorporated
visual information, which poses a significant challenge. Furthermore, existing
models that incorporate images in dialogue generation focus on discussing the
image itself. Our proposed approach presents a novel perspective on multi-modal
dialogue systems, which interprets the image in the context of the dialogue. By
doing so, we aim to expand the capabilities of current dialogue systems and
transition them from single modality (text) to multi-modality. However, there
is a lack of validated English datasets that contain both images and dialogue
contexts for this task. Thus, we propose a two-stage approach to automatically
construct a multi-modal dialogue dataset. In the first stage, we utilize
text-to-image similarity and sentence similarity to identify which utterances
could be replaced with an image. In the second stage, we replace those
utterances by selecting a subset of relevant images and filtering them with a
visual question answering model. We used this approach, along with additional
labeling, to create the IMage Augmented multi-modal Dialogue dataset (IMAD),
which can serve as a validated dataset for this task. Furthermore, we propose a
baseline model trained on this dataset, which outperforms model trained on the
same data without images and BlenderBot.

该研究提出了一种基于多模态视角的对话系统，并通过构建 IMAD 数据集和进行基线模型训练，证明了其在图像解释、数据集建设和模型表现方面具有潜在优势。

IMAD: 图像增强的多模态对话

IMAD: IMage-Augmented multi-modal Dialogue

Discovering social relations in images can make machines better interpret the
behavior of human beings. However, automatically recognizing social relations
in images is a challenging task due to the significant gap between the domains
of visual content and social relation. Existing studies separately process
various features such as faces expressions, body appearance, and contextual
objects, thus they cannot comprehensively capture the multi-granularity
semantics, such as scenes, regional cues of persons, and interactions among
persons and objects. To bridge the domain gap, we propose a Multi-Granularity
Reasoning framework for social relation recognition from images. The global
knowledge and mid-level details are learned from the whole scene and the
regions of persons and objects, respectively. Most importantly, we explore the
fine-granularity pose keypoints of persons to discover the interactions among
persons and objects. Specifically, the pose-guided Person-Object Graph and
Person-Pose Graph are proposed to model the actions from persons to object and
the interactions between paired persons, respectively. Based on the graphs,
social relation reasoning is performed by graph convolutional networks.
Finally, the global features and reasoned knowledge are integrated as a
comprehensive representation for social relation recognition. Extensive
experiments on two public datasets show the effectiveness of the proposed
framework.

本研究提出了一种基于多粒度推理的社交关系识别框架，包括全局知识，中层细节和细粒度姿态关键点。研究设计了基于姿态指导的人物对象图和人物姿态图来建模人物到对象的动作和配对人物之间的交互，并借助图卷积网络进行社交关系推理。实验结果表明提出的框架在两个公共数据集上具有较高的效果。

多粒度推理用于图像社交关系识别

Multi-Granularity Reasoning for Social Relation Recognition from Images

The idea of computer vision as the Bayesian inverse problem to computer
graphics has a long history and an appealing elegance, but it has proved
difficult to directly implement. Instead, most vision tasks are approached via
complex bottom-up processing pipelines. Here we show that it is possible to
write short, simple probabilistic graphics programs that define flexible
generative models and to automatically invert them to interpret real-world
images. Generative probabilistic graphics programs consist of a stochastic
scene generator, a renderer based on graphics software, a stochastic likelihood
model linking the renderer's output and the data, and latent variables that
adjust the fidelity of the renderer and the tolerance of the likelihood model.
Representations and algorithms from computer graphics, originally designed to
produce high-quality images, are instead used as the deterministic backbone for
highly approximate and stochastic generative models. This formulation combines
probabilistic programming, computer graphics, and approximate Bayesian
computation, and depends only on general-purpose, automatic inference
techniques. We describe two applications: reading sequences of degraded and
adversarially obscured alphanumeric characters, and inferring 3D road models
from vehicle-mounted camera images. Each of the probabilistic graphics programs
we present relies on under 20 lines of probabilistic code, and supports
accurate, approximately Bayesian inferences about ambiguous real-world images.

这篇论文介绍了一个基于概率编程和计算机图形学的方法来解决计算机视觉中的图形逆问题，提出了一种简洁的概率图形程序来定义灵活的生成模型，并自动反演真实世界图像以诠释它们。