In recent years, vision language pre-training frameworks have made
significant progress in natural language processing and computer vision,
achieving remarkable performance improvement on various downstream tasks.
However, when extended to point cloud data, existing works mainly focus on
building task-specific models, and fail to extract universal 3D vision-language
embedding that generalize well. We carefully investigate three common tasks in
semantic 3D scene understanding, and derive key insights into the development
of a pre-training model. Motivated by these observations, we propose a
vision-language pre-training framework 3DVLP (3D vision-language pre-training
with object contrastive learning), which transfers flexibly on 3D
vision-language downstream tasks. 3DVLP takes visual grounding as the proxy
task and introduces Object-level IoU-guided Detection (OID) loss to obtain
high-quality proposals in the scene. Moreover, we design Object-level
Cross-Contrastive alignment (OCC) task and Object-level Self-Contrastive
learning (OSC) task to align the objects with descriptions and distinguish
different objects in the scene, respectively. Extensive experiments verify the
excellent performance of 3DVLP on three 3D vision-language tasks, reflecting
its superiority in semantic 3D scene understanding.

本文提出了一种 3D 视觉语言预训练框架 3DVLP，可以在 3D 视觉语言下游任务中有很好的表现，该框架考虑了场景中物体的关联性，提出了多个任务来实现对象级交叉对齐和区分，与任务特定方法相比具有更好的泛化性能。