An effective pre-training framework with universal 3D representations is
extremely desired in perceiving large-scale dynamic scenes. However,
establishing such an ideal framework that is both task-generic and
label-efficient poses a challenge in unifying the representation of the same
primitive across diverse scenes. The current contrastive 3D pre-training
methods typically follow a frame-level consistency, which focuses on the 2D-3D
relationships in each detached image. Such inconsiderate consistency greatly
hampers the promising path of reaching an universal pre-training framework: (1)
The cross-scene semantic self-conflict, i.e., the intense collision between
primitive segments of the same semantics from different scenes; (2) Lacking a
globally unified bond that pushes the cross-scene semantic consistency into 3D
representation learning. To address above challenges, we propose a CSC
framework that puts a scene-level semantic consistency in the heart, bridging
the connection of the similar semantic segments across various scenes. To
achieve this goal, we combine the coherent semantic cues provided by the vision
foundation model and the knowledge-rich cross-scene prototypes derived from the
complementary multi-modality information. These allow us to train a universal
3D pre-training model that facilitates various downstream tasks with less
fine-tuning efforts. Empirically, we achieve consistent improvements over SOTA
pre-training approaches in semantic segmentation (+1.4% mIoU), object detection
(+1.0% mAP), and panoptic segmentation (+3.0% PQ) using their task-specific 3D
network on nuScenes. Code is released at this https URL,
hoping to inspire future research.

通过构建一个 CSC 框架，我们综合使用视觉基础模型提供的语义线索和多模态信息的知识丰富的跨场景原型，实现了一种具有各种下游任务的更少微调工作量的通用 3D 预训练模型。