Cross-modality registration between 2D images from cameras and 3D point
clouds from LiDARs is a crucial task in computer vision and robotic. Previous
methods estimate 2D-3D correspondences by matching point and pixel patterns
learned by neural networks, and use Perspective-n-Points (PnP) to estimate
rigid transformation during post-processing. However, these methods struggle to
map points and pixels to a shared latent space robustly since points and pixels
have very different characteristics with patterns learned in different manners
(MLP and CNN), and they also fail to construct supervision directly on the
transformation since the PnP is non-differentiable, which leads to unstable
registration results. To address these problems, we propose to learn a
structured cross-modality latent space to represent pixel features and 3D
features via a differentiable probabilistic PnP solver. Specifically, we design
a triplet network to learn VoxelPoint-to-Pixel matching, where we represent 3D
elements using both voxels and points to learn the cross-modality latent space
with pixels. We design both the voxel and pixel branch based on CNNs to operate
convolutions on voxels/pixels represented in grids, and integrate an additional
point branch to regain the information lost during voxelization. We train our
framework end-to-end by imposing supervisions directly on the predicted pose
distribution with a probabilistic PnP solver. To explore distinctive patterns
of cross-modality features, we design a novel loss with adaptive-weighted
optimization for cross-modality feature description. The experimental results
on KITTI and nuScenes datasets show significant improvements over the
state-of-the-art methods. The code and models are available at
this https URL

通过可微分的概率 PnP 求解器学习结构化的跨模态潜空间，以表示像素特征和 3D 特征，并通过 VoxelPoint-to-Pixel 匹配设计了三元组网络，该网络使用像素和点来学习潜空间。在 KITTI 和 nuScenes 数据集上的实验结果显示了与最先进方法相比的显著改进。

图像和 LiDAR 点云的可微分配准：体素点对应

Differentiable Registration of Images and LiDAR Point Clouds with  VoxelPoint-to-Pixel Matching

This paper presents DeepI2P: a novel approach for cross-modality registration
between an image and a point cloud. Given an image (e.g. from a rgb-camera) and
a general point cloud (e.g. from a 3D Lidar scanner) captured at different
locations in the same scene, our method estimates the relative rigid
transformation between the coordinate frames of the camera and Lidar. Learning
common feature descriptors to establish correspondences for the registration is
inherently challenging due to the lack of appearance and geometric correlations
across the two modalities. We circumvent the difficulty by converting the
registration problem into a classification and inverse camera projection
optimization problem. A classification neural network is designed to label
whether the projection of each point in the point cloud is within or beyond the
camera frustum. These labeled points are subsequently passed into a novel
inverse camera projection solver to estimate the relative pose. Extensive
experimental results on Oxford Robotcar and KITTI datasets demonstrate the
feasibility of our approach. Our source code is available at
this https URL

本文提出了 DeepI2P：一种新的图像和点云之间的跨模态配准方法。通过将配准问题转化为分类和逆投影优化问题，并使用分类神经网络来对点云中的每个点进行分类。通过这种方法估计相对刚性变换，进而解决了图像和点云之间特征描述缺失的问题。实验结果表明了我们方法的可行性。