Phrase Grounding aims to detect and localize objects in images that are referred to and are queried by natural language phrases. Phrase grounding finds applications in tasks such as Visual Dialog, Visual Search and Image-text co-reference resolution. In this paper, we present a framework that leverages information such as phrase category, relationships among neighboring phrases in a sentence and context to improve the performance of phrase grounding systems. We propose three modules: Proposal Indexing Network(PIN); Inter-phrase Regression Network(IRN) and Proposal Ranking Network(PRN) each of which analyze the region proposals of an image at increasing levels of detail by incorporating the above information. Also, in the absence of ground-truth spatial locations of the phrases(weakly-supervised), we propose knowledge transfer mechanisms that leverages the framework of PIN module. We demonstrate the effectiveness of our approach on the Flickr 30k Entities and ReferItGame datasets, for which we achieve improvements over state-of-the-art approaches in both supervised and weakly-supervised variants.

本文提出了一种利用自然语言短语的分类和周围短语的关系、上下文等信息，通过三个模块分析图像中的区域提案，用于检测自然语言短语指代的图像中的对象位置，同时提出了针对弱监督学习的知识转移机制，该方法通过在Flickr 30k Entities and ReferItGame数据集上的实验，在自监督和非自监督下均实现了与现有方法相比的改进。

PIRC Net：利用提案索引、关系和语境进行短语定位