Multi-view camera-based 3D detection is a challenging problem in computer
vision. Recent works leverage a pretrained LiDAR detection model to transfer
knowledge to a camera-based student network. However, we argue that there is a
major domain gap between the LiDAR BEV features and the camera-based BEV
features, as they have different characteristics and are derived from different
sources. In this paper, we propose Geometry Enhanced Masked Image Modeling
(GeoMIM) to transfer the knowledge of the LiDAR model in a pretrain-finetune
paradigm for improving the multi-view camera-based 3D detection. GeoMIM is a
multi-camera vision transformer with Cross-View Attention (CVA) blocks that
uses LiDAR BEV features encoded by the pretrained BEV model as learning
targets. During pretraining, GeoMIM's decoder has a semantic branch completing
dense perspective-view features and the other geometry branch reconstructing
dense perspective-view depth maps. The depth branch is designed to be
camera-aware by inputting the camera's parameters for better transfer
capability. Extensive results demonstrate that GeoMIM outperforms existing
methods on nuScenes benchmark, achieving state-of-the-art performance for
camera-based 3D object detection and 3D segmentation.

本文提出了一种名为 GeoMIM 的多相机视觉转换器，通过预训练 - 微调的方法将 LiDAR BEV 模型的知识传递给 GeoMIM，以改善多视图基于相机的三维检测，并在 nuscenes 基准测试中取得了最新的表现。