Referring expressions are natural language descriptions that identify a
particular object within a scene and are widely used in our daily
conversations. In this work, we focus on segmenting the object in an image
specified by a referring expression. To this end, we propose an end-to-end
trainable comprehension network that consists of the language and visual
encoders to extract feature representations from both domains. We introduce the
spatial-aware dynamic filters to transfer knowledge from text to image, and
effectively capture the spatial information of the specified object. To better
communicate between the language and visual modules, we employ a caption
generation network that takes features shared across both domains as input, and
improves both representations via a consistency that enforces the generated
sentence to be similar to the given referring expression. We evaluate the
proposed framework on two referring expression datasets and show that our
method performs favorably against the state-of-the-art algorithms.

本文介绍了一种通过端到端可训练的理解网络，由语言和视觉编码器组成，从语言和图像领域提取特征表示，提出了空间感知动态滤波器来转移文本到图像的知识并有效捕获指定对象的空间信息，并采用生成的字幕网络来加强语言和视觉模块之间的通信以及改进两者的表示，在两个数据集上评估了所提出的框架，并表明该方法在状态 - of-the-art 算法中表现良好。