Traditional 3D segmentation methods can only recognize a fixed range of
classes that appear in the training set, which limits their application in
real-world scenarios due to the lack of generalization ability. Large-scale
visual-language pre-trained models, such as CLIP, have shown their
generalization ability in the zero-shot 2D vision tasks, but are still unable
to be applied to 3D semantic segmentation directly. In this work, we focus on
zero-shot point cloud semantic segmentation and propose a simple yet effective
baseline to transfer the visual-linguistic knowledge implied in CLIP to point
cloud encoder at both feature and output levels. Both feature-level and
output-level alignments are conducted between 2D and 3D encoders for effective
knowledge transfer. Concretely, a Multi-granularity Cross-modal Feature
Alignment (MCFA) module is proposed to align 2D and 3D features from global
semantic and local position perspectives for feature-level alignment. For the
output level, per-pixel pseudo labels of unseen classes are extracted using the
pre-trained CLIP model as supervision for the 3D segmentation model to mimic
the behavior of the CLIP image encoder. Extensive experiments are conducted on
two popular benchmarks of point cloud segmentation. Our method outperforms
significantly previous state-of-the-art methods under zero-shot setting (+29.2%
mIoU on SemanticKITTI and 31.8% mIoU on nuScenes), and further achieves
promising results in the annotation-free point cloud semantic segmentation
setting, showing its great potential for label-efficient learning.

本文提出了一种简单而有效的基线方法，将 CLIP 中的视觉语言知识在特征和输出层面上转移到点云编码器，从而在零样本点云语义分割中取得了显著的性能提升，并在无标注点云语义分割设置中取得了有希望的结果，展示了其在标签效率学习方面的巨大潜力。