For robots to understand human instructions and perform meaningful tasks in
the near future, it is important to develop learned models that comprehend
referential language to identify common objects in real-world 3D scenes. In
this paper, we introduce a spatial-language model for a 3D visual grounding
problem. Specifically, given a reconstructed 3D scene in the form of point
clouds with 3D bounding boxes of potential object candidates, and a language
utterance referring to a target object in the scene, our model successfully
identifies the target object from a set of potential candidates. Specifically,
LanguageRefer uses a transformer-based architecture that combines spatial
embedding from bounding boxes with fine-tuned language embeddings from
DistilBert to predict the target object. We show that it performs competitively
on visio-linguistic datasets proposed by ReferIt3D. Further, we analyze its
spatial reasoning task performance decoupled from perception noise, the
accuracy of view-dependent utterances, and viewpoint annotations for potential
robotics applications.

提出了一种空间语言模型用于 3D 视觉定位问题，使用基于 Transformer 的架构将空间嵌入和 DistilBert 的语言嵌入结合起来进行目标对象预测，能够在 ReferIt3D 提出的数据集上表现出竞争性，可以被应用于机器人等领域的视觉任务中。