Multi-camera 3D object detection for autonomous driving is a challenging
problem that has garnered notable attention from both academia and industry. An
obstacle encountered in vision-based techniques involves the precise extraction
of geometry-conscious features from RGB images. Recent approaches have utilized
geometric-aware image backbones pretrained on depth-relevant tasks to acquire
spatial information. However, these approaches overlook the critical aspect of
view transformation, resulting in inadequate performance due to the
misalignment of spatial knowledge between the image backbone and view
transformation. To address this issue, we propose a novel geometric-aware
pretraining framework called GAPretrain. Our approach incorporates spatial and
structural cues to camera networks by employing the geometric-rich modality as
guidance during the pretraining phase. The transference of modal-specific
attributes across different modalities is non-trivial, but we bridge this gap
by using a unified bird's-eye-view (BEV) representation and structural hints
derived from LiDAR point clouds to facilitate the pretraining process.
GAPretrain serves as a plug-and-play solution that can be flexibly applied to
multiple state-of-the-art detectors. Our experiments demonstrate the
effectiveness and generalization ability of the proposed method. We achieve
46.2 mAP and 55.5 NDS on the nuScenes val set using the BEVFormer method, with
a gain of 2.7 and 2.1 points, respectively. We also conduct experiments on
various image backbones and view transformations to validate the efficacy of
our approach. Code will be released at
this https URL

提出了一种名为 GAPretrain 的几何意识预训练框架，通过使用几何丰富的模态和来自 LiDAR 点云的结构提示，实现跨不同模态之间的属性转移，从而解决多镜头自动驾驶中图像和视角不一致的问题，实现了很好的效果。