This paper introduces a novel self-supervised learning framework for
enhancing 3D perception in autonomous driving scenes. Specifically, our
approach, named NCLR, focuses on 2D-3D neural calibration, a novel pretext task
that estimates the rigid transformation aligning camera and LiDAR coordinate
systems. First, we propose the learnable transformation alignment to bridge the
domain gap between image and point cloud data, converting features into a
unified representation space for effective comparison and matching. Second, we
identify the overlapping area between the image and point cloud with the fused
features. Third, we establish dense 2D-3D correspondences to estimate the rigid
transformation. The framework not only learns fine-grained matching from points
to pixels but also achieves alignment of the image and point cloud at a
holistic level, understanding their relative pose. We demonstrate NCLR's
efficacy by applying the pre-trained backbone to downstream tasks, such as
LiDAR-based 3D semantic segmentation, object detection, and panoptic
segmentation. Comprehensive experiments on various datasets illustrate the
superiority of NCLR over existing self-supervised methods. The results confirm
that joint learning from different modalities significantly enhances the
network's understanding abilities and effectiveness of learned representation.
Code will be available at https://github.com/Eaphan/NCLR.

这篇论文介绍了一种新颖的自监督学习框架，用于增强自动驾驶场景中的 3D 感知。该方法名为 NCLR，专注于 2D-3D 神经校准，一种估计相机和 LiDAR 坐标系统对齐的创新前提任务。通过学习可训练的转换对齐，我们建立了图像和点云数据之间的领域差距，将特征转换成统一的表示空间，以进行有效的比较和匹配，从而实现了图像和点云的整体级别对齐。我们还通过将预训练的骨干网络应用于 LiDAR 的 3D 语义分割、目标检测和泛光分割等下游任务，证明了 NCLR 的有效性。在各种数据集上进行的综合实验证明了 NCLR 相对于现有自监督方法的优越性，并且结果表明，不同模态的联合学习显著提高了网络的理解能力和学习表示的有效性。