An LBYL (`Look Before You Leap') Network is proposed for end-to-end trainable
one-stage visual grounding. The idea behind LBYL-Net is intuitive and
straightforward: we follow a language's description to localize the target
object based on its relative spatial relation to `Landmarks', which is
characterized by some spatial positional words and some descriptive words about
the object. The core of our LBYL-Net is a landmark feature convolution module
that transmits the visual features with the guidance of linguistic description
along with different directions. Consequently, such a module encodes the
relative spatial positional relations between the current object and its
context. Then we combine the contextual information from the landmark feature
convolution module with the target's visual features for grounding. To make
this landmark feature convolution light-weight, we introduce a dynamic
programming algorithm (termed dynamic max pooling) with low complexity to
extract the landmark feature. Thanks to the landmark feature convolution
module, we mimic the human behavior of `Look Before You Leap' to design an
LBYL-Net, which takes full consideration of contextual information. Extensive
experiments show our method's effectiveness in four grounding datasets.
Specifically, our LBYL-Net outperforms all state-of-the-art two-stage and
one-stage methods on ReferitGame. On RefCOCO and RefCOCO+, Our LBYL-Net also
achieves comparable results or even better results than existing one-stage
methods.

提出了一种 LBYL（“先看再跳”）网络，用于端到端可训练的单阶段视觉定位，其核心是地标特征卷积模块，以指导语言描述的方式传递视觉特征，并结合与目标的上下文信息进行定位，实验结果表明，LBYL-Net 在 ReferitGame 上优于所有现有的两阶段和一阶段方法，在 RefCOCO 和 RefCOCO + 上的表现也与现有的一阶段方法相当甚至更好。

先斩后奏：一阶段视觉定位中学习地标特征

Look Before You Leap: Learning Landmark Features for One-Stage Visual  Grounding

We improve one-stage visual grounding by addressing current limitations on
grounding long and complex queries. Existing one-stage methods encode the
entire language query as a single sentence embedding vector, e.g., taking the
embedding from BERT or the hidden state from LSTM. This single vector
representation is prone to overlooking the detailed descriptions in the query.
To address this query modeling deficiency, we propose a recursive sub-query
construction framework, which reasons between image and query for multiple
rounds and reduces the referring ambiguity step by step. We show our new
one-stage method obtains 5.0%, 4.5%, 7.5%, 12.8% absolute improvements over the
state-of-the-art one-stage baseline on ReferItGame, RefCOCO, RefCOCO+, and
RefCOCOg, respectively. In particular, superior performances on longer and more
complex queries validates the effectiveness of our query modeling.

提出一种递归子查询构建框架，解决当前一阶段视觉基础的限制，提高了长而复杂查询的精度，效果比现有一阶段基线模型在多个基准数据集上都有显著的提高。