3D visual grounding aims at grounding a natural language description about a
3D scene, usually represented in the form of 3D point clouds, to the targeted
object region. Point clouds are sparse, noisy, and contain limited semantic
information compared with 2D images. These inherent limitations make the 3D
visual grounding problem more challenging. In this study, we propose 2D
Semantics Assisted Training (SAT) that utilizes 2D image semantics in the
training stage to ease point-cloud-language joint representation learning and
assist 3D visual grounding. The main idea is to learn auxiliary alignments
between rich, clean 2D object representations and the corresponding objects or
mentioned entities in 3D scenes. SAT takes 2D object semantics, i.e., object
label, image feature, and 2D geometric feature, as the extra input in training
but does not require such inputs during inference. By effectively utilizing 2D
semantics in training, our approach boosts the accuracy on the Nr3D dataset
from 37.7% to 49.2%, which significantly surpasses the non-SAT baseline with
the identical network architecture and inference input. Our approach
outperforms the state of the art by large margins on multiple 3D visual
grounding datasets, i.e., +10.4% absolute accuracy on Nr3D, +9.9% on Sr3D, and
+5.6% on ScanRef.

该研究提出了 2D Semantics Assisted Training（SAT）来提高 3D 视觉定位的准确性，通过学习丰富，干净的 2D 对象表示与 3D 场景中的对象或提到的实体之间的对齐关系，辅助 3D 视觉定位，将 2D 语义有效地利用于训练，使方法在 3D 视觉定位数据集上的准确性从 37.7％提高到 49.2％，并且在多个 3D 视觉定位数据集上均优于同类方法。