In this paper, we present a neat yet effective transformer-based framework
for visual grounding, namely TransVG, to address the task of grounding a
language query to the corresponding region onto an image. The state-of-the-art
methods, including two-stage or one-stage ones, rely on a complex module with
manually-designed mechanisms to perform the query reasoning and multi-modal
fusion. However, the involvement of certain mechanisms in fusion module design,
such as query decomposition and image scene graph, makes the models easily
overfit to datasets with specific scenarios, and limits the plenitudinous
interaction between the visual-linguistic context. To avoid this caveat, we
propose to establish the multi-modal correspondence by leveraging transformers,
and empirically show that the complex fusion modules e.g., modular attention
network, dynamic graph, and multi-modal tree) can be replaced by a simple stack
of transformer encoder layers with higher performance. Moreover, we
re-formulate the visual grounding as a direct coordinates regression problem
and avoid making predictions out of a set of candidates i.e., region proposals
or anchor boxes). Extensive experiments are conducted on five widely used
datasets, and a series of state-of-the-art records are set by our TransVG. We
build the benchmark of transformer-based visual grounding framework and make
the code available at https://github.com/djiajunustc/TransVG.

本文提出了一种基于 Transformer 的视觉定位框架 TransVG，通过建立多模态对应关系，实现使用简单 Transformer 编码器层替代复杂的融合模块，在五个广泛使用的数据集上取得了一系列最先进的记录并构建了基于 Transformer 的视觉定位框架基准。