Contrastive Language-Image Pre-training (CLIP) has shown promising open-world
performance on 2D image tasks, while its transferred capacity on 3D point
clouds, i.e., PointCLIP, is still far from satisfactory. In this work, we
propose PointCLIP V2, a powerful 3D open-world learner, to fully unleash the
potential of CLIP on 3D point cloud data. First, we introduce a realistic shape
projection module to generate more realistic depth maps for CLIP's visual
encoder, which is quite efficient and narrows the domain gap between projected
point clouds with natural images. Second, we leverage large-scale language
models to automatically design a more descriptive 3D-semantic prompt for CLIP's
textual encoder, instead of the previous hand-crafted one. Without introducing
any training in 3D domains, our approach significantly surpasses PointCLIP by
+42.90%, +40.44%, and +28.75% accuracy on three datasets for zero-shot 3D
classification. Furthermore, PointCLIP V2 can be extended to few-shot
classification, zero-shot part segmentation, and zero-shot 3D object detection
in a simple manner, demonstrating our superior generalization ability for 3D
open-world learning. Code will be available at
this https URL

本文介绍了一种名为 PointCLIP V2 的 3D 开放世界学习器，它使用对比语言 - 图像预训练（CLIP）以及大规模语言模型来在 3D 点云数据上进行零 - shot 分类、部分分割和物体检测。PointCLIP V2 通过引入现实形状投影模块和自动设计更具描述性的 3D - 语义暗示，显著超越了 PointCLIP 的性能。