Large-scale vision 2D vision language models, such as CLIP can be aligned
with a 3D encoder to learn generalizable (open-vocabulary) 3D vision models.
However, current methods require supervised pre-training for such alignment,
and the performance of such 3D zero-shot models remains sub-optimal for
real-world adaptation. In this work, we propose an optimization framework:
Cross-MoST: Cross-Modal Self-Training, to improve the label-free classification
performance of a zero-shot 3D vision model by simply leveraging unlabeled 3D
data and their accompanying 2D views. We propose a student-teacher framework to
simultaneously process 2D views and 3D point clouds and generate joint pseudo
labels to train a classifier and guide cross-model feature alignment. Thereby
we demonstrate that 2D vision language models such as CLIP can be used to
complement 3D representation learning to improve classification performance
without the need for expensive class annotations. Using synthetic and
real-world 3D datasets, we further demonstrate that Cross-MoST enables
efficient cross-modal knowledge exchange resulting in both image and point
cloud modalities learning from each other's rich representations.

通过使用 Cross-MoST 优化框架，结合 CLIP 等 2D 视觉语言模型，可以提高无需标签的零样本 3D 视觉模型的分类性能，并且实现图像和点云模态之间的跨模态知识交流。