Discovering social relations in images can make machines better interpret the
behavior of human beings. However, automatically recognizing social relations
in images is a challenging task due to the significant gap between the domains
of visual content and social relation. Existing studies separately process
various features such as faces expressions, body appearance, and contextual
objects, thus they cannot comprehensively capture the multi-granularity
semantics, such as scenes, regional cues of persons, and interactions among
persons and objects. To bridge the domain gap, we propose a Multi-Granularity
Reasoning framework for social relation recognition from images. The global
knowledge and mid-level details are learned from the whole scene and the
regions of persons and objects, respectively. Most importantly, we explore the
fine-granularity pose keypoints of persons to discover the interactions among
persons and objects. Specifically, the pose-guided Person-Object Graph and
Person-Pose Graph are proposed to model the actions from persons to object and
the interactions between paired persons, respectively. Based on the graphs,
social relation reasoning is performed by graph convolutional networks.
Finally, the global features and reasoned knowledge are integrated as a
comprehensive representation for social relation recognition. Extensive
experiments on two public datasets show the effectiveness of the proposed
framework.

本研究提出了一种基于多粒度推理的社交关系识别框架，包括全局知识，中层细节和细粒度姿态关键点。研究设计了基于姿态指导的人物对象图和人物姿态图来建模人物到对象的动作和配对人物之间的交互，并借助图卷积网络进行社交关系推理。实验结果表明提出的框架在两个公共数据集上具有较高的效果。