Using only image-sentence pairs, weakly-supervised visual-textual grounding
aims to learn region-phrase correspondences of the respective entity mentions.
Compared to the supervised approach, learning is more difficult since bounding
boxes and textual phrases correspondences are unavailable. In light of this, we
propose the Semantic Prior Refinement Model (SPRM), whose predictions are
obtained by combining the output of two main modules. The first untrained
module aims to return a rough alignment between textual phrases and bounding
boxes. The second trained module is composed of two sub-components that refine
the rough alignment to improve the accuracy of the final phrase-bounding box
alignments. The model is trained to maximize the multimodal similarity between
an image and a sentence, while minimizing the multimodal similarity of the same
sentence and a new unrelated image, carefully selected to help the most during
training. Our approach shows state-of-the-art results on two popular datasets,
Flickr30k Entities and ReferIt, shining especially on ReferIt with a 9.6%
absolute improvement. Moreover, thanks to the untrained component, it reaches
competitive performances just using a small fraction of training examples.

论文提出了一种基于弱化监督的视觉文本对齐模型 SPRM，通过组合输出的两个模块的预测结果学习文本短语和边界框之间的对应关系，达到了最先进的实验效果，并且在使用少量训练样例时也具有竞争性能。