Compared with the visual grounding on 2D images, the natural-language-guided
3D object localization on point clouds is more challenging. In this paper, we
propose a new model, named InstanceRefer, to achieve a superior 3D visual
grounding through the grounding-by-matching strategy. In practice, our model
first predicts the target category from the language descriptions using a
simple language classification model. Then, based on the category, our model
sifts out a small number of instance candidates (usually less than 20) from the
panoptic segmentation of point clouds. Thus, the non-trivial 3D visual
grounding task has been effectively re-formulated as a simplified
instance-matching problem, considering that instance-level candidates are more
rational than the redundant 3D object proposals. Subsequently, for each
candidate, we perform the multi-level contextual inference, i.e., referring
from instance attribute perception, instance-to-instance relation perception,
and instance-to-background global localization perception, respectively.
Eventually, the most relevant candidate is selected and localized by ranking
confidence scores, which are obtained by the cooperative holistic
visual-language feature matching. Experiments confirm that our method
outperforms previous state-of-the-arts on ScanRefer online benchmark and
Nr3D/Sr3D datasets.

本文提出了一种名为 InstanceRefer 的模型，该模型利用语言描述实现目标类别预测，从 3D 点云的广义实例分割中筛选出少量实例候选项，利用多级语境推理和多层次特征匹配来实现最相关候选项的选择和定位，从而实现了优越的 3D 视觉定位结果