We propose a weakly-supervised approach that takes image-sentence pairs as input and learns to visually ground (i.e., localize) arbitrary linguistic phrases, in the form of spatial attention masks. Specifically, the model is trained with images and their associated image-level captions, without any explicit region-to-phrase correspondence annotations. To this end, we introduce an end-to-end model which learns visual groundings of phrases with two types of carefully designed loss functions. In addition to the standard discriminative loss, which enforces that attended image regions and phrases are consistently encoded, we propose a novel structural loss which makes use of the parse tree structures induced by the sentences. In particular, we ensure complementarity among the attention masks that correspond to sibling noun phrases, and compositionality of attention masks among the children and parent phrases, as defined by the sentence parse tree. We validate the effectiveness of our approach on the Microsoft COCO and Visual Genome datasets.

文章提出了一种弱监督方法，该方法以图像-句子对为输入，学习以空间注意力掩模的形式定位任意语言短语。我们介绍了一种端到端模型，使用两种精心设计的损失函数学习短语的视觉基础，并在句子中使用引诱的解析树结构来确保注意力掩码的补充性及组合性。

具有语言结构的短语弱监督视觉定位