Visual grounding, which aims to build a correspondence between visual objects
and their language entities, plays a key role in cross-modal scene
understanding. One promising and scalable strategy for learning visual
grounding is to utilize weak supervision from only image-caption pairs.
Previous methods typically rely on matching query phrases directly to a
precomputed, fixed object candidate pool, which leads to inaccurate
localization and ambiguous matching due to lack of semantic relation
constraints.
In our paper, we propose a novel context-aware weakly-supervised learning
method that incorporates coarse-to-fine object refinement and entity relation
modeling into a two-stage deep network, capable of producing more accurate
object representation and matching. To effectively train our network, we
introduce a self-taught regression loss for the proposal locations and a
classification loss based on parsed entity relations.
Extensive experiments on two public benchmarks Flickr30K Entities and
ReferItGame demonstrate the efficacy of our weakly grounding framework. The
results show that we outperform the previous methods by a considerable margin,
achieving 59.27\% top-1 accuracy in Flickr30K Entities and 37.68\% in the
ReferItGame dataset respectively (Code is available at
this https URL).

本研究提出了一种新的上下文感知弱监督学习方法，它将粗到细的物体细化和实体关系建模结合到一个两阶段深度网络中，以更准确地表述和匹配物体，并通过自我学习回归和关系分析来有效训练必要的分类方法。在 Flickr30K 和 ReferItGame 数据集上的广泛实验表明，本文所提出的弱强化框架比以前的方法具有更好的算法性能，Flickr30K 实体和 ReferItGame 数据集上的 Top-1 准确度分别达到 59.27％和 37.68％。