This paper presents DeepI2P: a novel approach for cross-modality registration between an image and a point cloud. Given an image (e.g. from a rgb-camera) and a general point cloud (e.g. from a 3D Lidar scanner) captured at different locations in the same scene, our method estimates the relative rigid transformation between the coordinate frames of the camera and Lidar. Learning common feature descriptors to establish correspondences for the registration is inherently challenging due to the lack of appearance and geometric correlations across the two modalities. We circumvent the difficulty by converting the registration problem into a classification and inverse camera projection optimization problem. A classification neural network is designed to label whether the projection of each point in the point cloud is within or beyond the camera frustum. These labeled points are subsequently passed into a novel inverse camera projection solver to estimate the relative pose. Extensive experimental results on Oxford Robotcar and KITTI datasets demonstrate the feasibility of our approach. Our source code is available at https://github.com/lijx10/DeepI2P

本文提出了DeepI2P：一种新的图像和点云之间的跨模态配准方法。通过将配准问题转化为分类和逆投影优化问题，并使用分类神经网络来对点云中的每个点进行分类。通过这种方法估计相对刚性变换，进而解决了图像和点云之间特征描述缺失的问题。实验结果表明了我们方法的可行性。

DeepI2P: 基于深度分类的图像到点云注册