Learning robust and scalable visual representations from massive multi-view
video data remains a challenge in computer vision and autonomous driving.
Existing pre-training methods either rely on expensive supervised learning with
3D annotations, limiting the scalability, or focus on single-frame or monocular
inputs, neglecting the temporal information. We propose MIM4D, a novel
pre-training paradigm based on dual masked image modeling (MIM). MIM4D
leverages both spatial and temporal relations by training on masked multi-view
video inputs. It constructs pseudo-3D features using continuous scene flow and
projects them onto 2D plane for supervision. To address the lack of dense 3D
supervision, MIM4D reconstruct pixels by employing 3D volumetric differentiable
rendering to learn geometric representations. We demonstrate that MIM4D
achieves state-of-the-art performance on the nuScenes dataset for visual
representation learning in autonomous driving. It significantly improves
existing methods on multiple downstream tasks, including BEV segmentation (8.7%
IoU), 3D object detection (3.5% mAP), and HD map construction (1.4% mAP). Our
work offers a new choice for learning representation at scale in autonomous
driving. Code and models are released at this https URL

从大规模多视角视频数据中学习强大且可扩展的视觉表征仍然是计算机视觉和自动驾驶领域的一项挑战。我们提出了 MIM4D，一种基于双重遮罩图像建模（MIM）的新的预训练范例，它通过训练遮罩的多视角视频输入来利用空间和时间关系，从而构建伪 3D 特征并进行监督学习，以解决缺乏密集 3D 监测的问题。通过采用 3D 体积可微分渲染来学习几何表征，MIM4D 在自动驾驶中的可视表征学习任务中取得了最先进的性能，显著提高了多个下游任务的表现。