Training models to apply common-sense linguistic knowledge and visual
concepts from 2D images to 3D scene understanding is a promising direction that
researchers have only recently started to explore. However, it still remains
understudied whether 2D distilled knowledge can provide useful representations
for downstream 3D vision-language tasks such as 3D question answering. In this
paper, we propose a novel 3D pre-training Vision-Language method, namely
Multi-CLIP, that enables a model to learn language-grounded and transferable 3D
scene point cloud representations. We leverage the representational power of
the CLIP model by maximizing the agreement between the encoded 3D scene
features and the corresponding 2D multi-view image and text embeddings in the
CLIP space via a contrastive objective. To validate our approach, we consider
the challenging downstream tasks of 3D Visual Question Answering (3D-VQA) and
3D Situated Question Answering (3D-SQA). To this end, we develop novel
multi-modal transformer-based architectures and we demonstrate how our
pre-training method can benefit their performance. Quantitative and qualitative
experimental results show that Multi-CLIP outperforms state-of-the-art works
across the downstream tasks of 3D-VQA and 3D-SQA and leads to a well-structured
3D scene feature space.

本研究提出一种名为 MULTI-CLIP 的 3D 预训练视觉语言模型，可有效提高现有 3D 视觉问答任务的表现并构建出具有良好结构的 3D 场景特征空间。