Training models to apply linguistic knowledge and visual concepts from 2D
images to 3D world understanding is a promising direction that researchers have
only recently started to explore. In this work, we design a novel 3D
pre-training Vision-Language method that helps a model learn semantically
meaningful and transferable 3D scene point cloud representations. We inject the
representational power of the popular CLIP model into our 3D encoder by
aligning the encoded 3D scene features with the corresponding 2D image and text
embeddings produced by CLIP. To assess our model's 3D world reasoning
capability, we evaluate it on the downstream task of 3D Visual Question
Answering. Experimental quantitative and qualitative results show that our
pre-training method outperforms state-of-the-art works in this task and leads
to an interpretable representation of 3D scene features.

本文介绍了一种新颖的 3D 预训练视觉 - 语言方法，将来自 2D 图像的语言知识和视觉概念应用于 3D 世界的理解，使用流行的 CLIP 模型代入编码的 3D 场景特征以评估其 3D 世界推理能力，并在 3D 视觉问答下游任务中证明了该方法的优越性和可解释性。