Textual grounding is an important but challenging task for human-computer
interaction, robotics and knowledge mining. Existing algorithms generally
formulate the task as selection from a set of bounding box proposals obtained
from deep net based systems. In this work, we demonstrate that we can cast the
problem of textual grounding into a unified framework that permits efficient
search over all possible bounding boxes. Hence, the method is able to consider
significantly more proposals and doesn't rely on a successful first stage
hypothesizing bounding box proposals. Beyond, we demonstrate that the trained
parameters of our model can be used as word-embeddings which capture
spatial-image relationships and provide interpretability. Lastly, at the time
of submission, our approach outperformed the current state-of-the-art methods
on the Flickr 30k Entities and the ReferItGame dataset by 3.08% and 7.77%
respectively.

本文研究了文本 grounding 的问题，提出了一种统一的框架来有效地搜索所有可能的 bounding box 提议，通过训练的模型参数作为 word-embedding，实现了空间 - 图像关系的捕捉和解释性，实验表明本方法在 Flickr 30k Entities 和 ReferItGame 数据集上的表现要好于当前最先进的方法。

可解释的、全局最优的基于图像概念的文本 grounding 预测

Interpretable and Globally Optimal Prediction for Textual Grounding  using Image Concepts

Grounding (i.e. localizing) arbitrary, free-form textual phrases in visual
content is a challenging problem with many applications for human-computer
interaction and image-text reference resolution. Few datasets provide the
ground truth spatial localization of phrases, thus it is desirable to learn
from data with no or little grounding supervision. We propose a novel approach
which learns grounding by reconstructing a given phrase using an attention
mechanism, which can be either latent or optimized directly. During training
our approach encodes the phrase using a recurrent network language model and
then learns to attend to the relevant image region in order to reconstruct the
input phrase. At test time, the correct attention, i.e., the grounding, is
evaluated. If grounding supervision is available it can be directly applied via
a loss over the attention mechanism. We demonstrate the effectiveness of our
approach on the Flickr 30k Entities and ReferItGame datasets with different
levels of supervision, ranging from no supervision over partial supervision to
full supervision. Our supervised variant improves by a large margin over the
state-of-the-art on both datasets.

通过采用注意力机制来重构给定的短语，本论文提出了一种新的接近无监督学习的方法来学习 grounding，该方法不需要太多的地面实时监督，有效提高了在 Flickr 30k 实体数据集上的表现。