Current successful methods of 3D scene perception rely on the large-scale
annotated point cloud, which is tedious and expensive to acquire. In this
paper, we propose Model2Scene, a novel paradigm that learns free 3D scene
representation from Computer-Aided Design (CAD) models and languages. The main
challenges are the domain gaps between the CAD models and the real scene's
objects, including model-to-scene (from a single model to the scene) and
synthetic-to-real (from synthetic model to real scene's object). To handle the
above challenges, Model2Scene first simulates a crowded scene by mixing
data-augmented CAD models. Next, we propose a novel feature regularization
operation, termed Deep Convex-hull Regularization (DCR), to project point
features into a unified convex hull space, reducing the domain gap. Ultimately,
we impose contrastive loss on language embedding and the point features of CAD
models to pre-train the 3D network. Extensive experiments verify the learned 3D
scene representation is beneficial for various downstream tasks, including
label-free 3D object salient detection, label-efficient 3D scene perception and
zero-shot 3D semantic segmentation. Notably, Model2Scene yields impressive
label-free 3D object salient detection with an average mAP of 46.08\% and
55.49\% on the ScanNet and S3DIS datasets, respectively. The code will be
publicly available.

当前成功的三维场景感知方法依赖于大规模标注的点云，获取费时费力。本文提出了 Model2Scene，一种新颖的范式，从计算机辅助设计 (CAD) 模型和语言中学习自由的三维场景表示。主要挑战是 CAD 模型与真实场景对象之间的领域差异，包括模型到场景的转换和合成模型到真实场景对象的转换。为了应对上述挑战，Model2Scene 首先通过混合数据增强的 CAD 模型模拟拥挤的场景。然后，我们提出了一种称为深凸包正则化 (DCR) 的新颖特征正则化操作，将点特征投影到统一的凸包空间，减小领域差距。最后，我们对 CAD 模型的语言嵌入和点特征进行对比损失的预训练，形成 3D 网络。大量实验证实了学习到的三维场景表示对各种下游任务的益处，包括无标签三维物体显著目标检测、标签高效三维场景感知和零样本三维语义分割。值得注意的是，Model2Scene 在 ScanNet 和 S3DIS 数据集上实现了令人印象深刻的无标签三维物体显著目标检测，平均 mAP 分别为 46.08% 和 55.49%。代码将公开提供。