Contrastive Language-Image Pre-training, benefiting from large-scale
unlabeled text-image pairs, has demonstrated great performance in open-world
vision understanding tasks. However, due to the limited Text-3D data pairs,
adapting the success of 2D Vision-Language Models (VLM) to the 3D space remains
an open problem. Existing works that leverage VLM for 3D understanding
generally resort to constructing intermediate 2D representations for the 3D
data, but at the cost of losing 3D geometry information. To take a step toward
open-world 3D vision understanding, we propose Contrastive Language-Image-Point
Cloud Pretraining (CLIP$^2$) to directly learn the transferable 3D point cloud
representation in realistic scenarios with a novel proxy alignment mechanism.
Specifically, we exploit naturally-existed correspondences in 2D and 3D
scenarios, and build well-aligned and instance-based text-image-point proxies
from those complex scenarios. On top of that, we propose a cross-modal
contrastive objective to learn semantic and instance-level aligned point cloud
representation. Experimental results on both indoor and outdoor scenarios show
that our learned 3D representation has great transfer ability in downstream
tasks, including zero-shot and few-shot 3D recognition, which boosts the
state-of-the-art methods by large margins. Furthermore, we provide analyses of
the capability of different representations in real scenarios and present the
optional ensemble scheme.

本文提出 Contrastive Language-Image-Point Cloud Pretraining (CLIP2) 方法，通过建立二三维场景对齐的实例文本 - 图像 - 点云代理和交叉模态对比学习策略，直接学习可迁移的三维点云表示，并在室内和室外场景中实验验证了其在零样本和少样本三维识别任务中的显著性能提升。