Foundation models have made significant strides in 2D and language tasks such
as image segmentation, object detection, and visual-language understanding.
Nevertheless, their potential to enhance 3D scene representation learning
remains largely untapped due to the domain gap. In this paper, we propose an
innovative methodology Bridge3D to address this gap, pre-training 3D models
using features, semantic masks, and captions sourced from foundation models.
Specifically, our approach utilizes semantic masks from these models to guide
the masking and reconstruction process in the masked autoencoder. This strategy
enables the network to concentrate more on foreground objects, thereby
enhancing 3D representation learning. Additionally, we bridge the 3D-text gap
at the scene level by harnessing image captioning foundation models. To further
facilitate knowledge distillation from well-learned 2D and text representations
to the 3D model, we introduce a novel method that employs foundation models to
generate highly accurate object-level masks and semantic text information at
the object level. Our approach notably outshines state-of-the-art methods in 3D
object detection and semantic segmentation tasks. For instance, on the ScanNet
dataset, our method surpasses the previous state-of-the-art method, PiMAE, by a
significant margin of 5.3%.

该论文介绍了一种名为 Bridge3D 的创新方法，通过预训练使用基础模型的特征、语义掩码和说明来预训练 3D 模型，从而增强 3D 场景表示学习，包括使用基础模型的语义掩码来指导掩码和重建过程。此外，作者还介绍了一种新方法，使用基础模型生成高精度的物体级掩码和语义文本信息，进一步促进基础 2D 和文本表示向 3D 模型的知识转移。该方法在 3D 对象检测和语义分割任务中优于现有的最先进方法，在 ScanNet 数据集上的最佳结果超过之前的最优方法 PiMAE 5.3%。