In autonomous driving, multi-modal perception models leveraging inputs from
multiple sensors exhibit strong robustness in degraded environments. However,
these models face challenges in efficiently and effectively transferring
learned representations across different modalities and tasks. This paper
presents NeRF-Supervised Masked Auto Encoder (NS-MAE), a self-supervised
pre-training paradigm for transferable multi-modal representation learning.
NS-MAE is designed to provide pre-trained model initializations for efficient
and high-performance fine-tuning. Our approach uses masked multi-modal
reconstruction in neural radiance fields (NeRF), training the model to
reconstruct missing or corrupted input data across multiple modalities.
Specifically, multi-modal embeddings are extracted from corrupted LiDAR point
clouds and images, conditioned on specific view directions and locations. These
embeddings are then rendered into projected multi-modal feature maps using
neural rendering techniques. The original multi-modal signals serve as
reconstruction targets for the rendered feature maps, facilitating
self-supervised representation learning. Extensive experiments demonstrate the
promising transferability of NS-MAE representations across diverse multi-modal
and single-modal perception models. This transferability is evaluated on
various 3D perception downstream tasks, such as 3D object detection and BEV map
segmentation, using different amounts of fine-tuning labeled data. Our code
will be released to support the community.

这篇论文介绍了一种用于可转移多模式表示学习的自监督预训练范式，利用 NeRF 支持的遮蔽自动编码器（NS-MAE）来提供高效且高性能的微调的预训练模型初始化，通过在神经辐射场（NeRF）中进行遮蔽多模式重建来训练模型以重建缺失或损坏的多模式输入数据，证明了 NS-MAE 表示在不同的多模式和单模式感知模型之间的良好可转移性，该可转移性在不同程度的微调标签数据下通过各种 3D 感知下游任务进行了评估，例如 3D 对象检测和 BEV 地图分割。