Grounding referring expressions in images aims to locate the object instance
in an image described by a referring expression. It involves a joint
understanding of natural language and image content, and is essential for a
range of visual tasks related to human-computer interaction. As a
language-to-vision matching task, the core of this problem is to not only
extract all the necessary information (i.e., objects and the relationships
among them) in both the image and referring expression, but also make full use
of context information to align cross-modal semantic concepts in the extracted
information. Unfortunately, existing work on grounding referring expressions
fails to accurately extract multi-order relationships from the referring
expression and associate them with the objects and their related contexts in
the image. In this paper, we propose a Cross-Modal Relationship Extractor
(CMRE) to adaptively highlight objects and relationships (spatial and semantic
relations) related to the given expression with a cross-modal attention
mechanism, and represent the extracted information as a language-guided visual
relation graph. In addition, we propose a Gated Graph Convolutional Network
(GGCN) to compute multimodal semantic contexts by fusing information from
different modes and propagating multimodal information in the structured
relation graph. Experimental results on three common benchmark datasets show
that our Cross-Modal Relationship Inference Network, which consists of CMRE and
GGCN, significantly surpasses all existing state-of-the-art methods. Code is
available at this https URL

本文提出了一种 Cross-Modal Relationship Extractor (CMRE) 的方法，利用跨模式的注意力机制自适应地提取和关联表达式中与对象相关的空间和语义关系，并将提取的信息表示为语言导向的视觉关系图。同时，我们还提出了一种 Gated Graph Convolutional Network (GGCN)，通过融合不同模式的信息并在结构化的关系图中传播多模式信息来计算多模式语义上下文，并在三个通用基准数据集上证明了该方法的显著优于所有现有的最新方法。