Recently proposed fine-grained 3D visual grounding is an essential and
challenging task, whose goal is to identify the 3D object referred by a natural
language sentence from other distractive objects of the same category. Existing
works usually adopt dynamic graph networks to indirectly model the
intra/inter-modal interactions, making the model difficult to distinguish the
referred object from distractors due to the monolithic representations of
visual and linguistic contents. In this work, we exploit Transformer for its
natural suitability on permutation-invariant 3D point clouds data and propose a
TransRefer3D network to extract entity-and-relation aware multimodal context
among objects for more discriminative feature learning. Concretely, we devise
an Entity-aware Attention (EA) module and a Relation-aware Attention (RA)
module to conduct fine-grained cross-modal feature matching. Facilitated by
co-attention operation, our EA module matches visual entity features with
linguistic entity features while RA module matches pair-wise visual relation
features with linguistic relation features, respectively. We further integrate
EA and RA modules into an Entity-and-Relation aware Contextual Block (ERCB) and
stack several ERCBs to form our TransRefer3D for hierarchical multimodal
context modeling. Extensive experiments on both Nr3D and Sr3D datasets
demonstrate that our proposed model significantly outperforms existing
approaches by up to 10.6% and claims the new state-of-the-art. To the best of
our knowledge, this is the first work investigating Transformer architecture
for fine-grained 3D visual grounding task.

本文提出了一种基于 Transformer 的 TransRefer3D 网络，利用实体和关系感知的多模态上下文提取模块，通过协同注意力操作进行跨模态匹配，构建了逐层的多模态上下文模型，从而在细粒度 3D 视觉定位任务中实现了学习出更具区分度的特征，并在实验中取得领先的性能。