The goal of open-vocabulary detection is to identify novel objects based on
arbitrary textual descriptions. In this paper, we address open-vocabulary 3D
point-cloud detection by a dividing-and-conquering strategy, which involves: 1)
developing a point-cloud detector that can learn a general representation for
localizing various objects, and 2) connecting textual and point-cloud
representations to enable the detector to classify novel object categories
based on text prompting. Specifically, we resort to rich image pre-trained
models, by which the point-cloud detector learns localizing objects under the
supervision of predicted 2D bounding boxes from 2D pre-trained detectors.
Moreover, we propose a novel de-biased triplet cross-modal contrastive learning
to connect the modalities of image, point-cloud and text, thereby enabling the
point-cloud detector to benefit from vision-language pre-trained
models,i.e.,CLIP. The novel use of image and vision-language pre-trained models
for point-cloud detectors allows for open-vocabulary 3D object detection
without the need for 3D annotations. Experiments demonstrate that the proposed
method improves at least 3.03 points and 7.47 points over a wide range of
baselines on the ScanNet and SUN RGB-D datasets, respectively. Furthermore, we
provide a comprehensive analysis to explain why our approach works.

本文探讨了通过分治策略进行开放式词汇 3D 点云检测的方法，该方法使用图像预训练模型来学习定位各种对象，并连接文本和点云表示，以便根据提示文本对新对象类别进行分类，从而实现开放式词汇 3D 目标检测，同时在 ScanNet 和 SUN RGB-D 数据集上实现了显著的性能提升。