Information inside visual and LiDAR data is well complementary derived from
the fine-grained texture of images and massive geometric information in point
clouds. However, it remains challenging to explore effective visual-LiDAR
fusion, mainly due to the intrinsic data structure inconsistency between two
modalities: Images are regular and dense, but LiDAR points are unordered and
sparse. To address the problem, we propose a local-to-global fusion network
with bi-directional structure alignment. To obtain locally fused features, we
project points onto image plane as cluster centers and cluster image pixels
around each center. Image pixels are pre-organized as pseudo points for
image-to-point structure alignment. Then, we convert points to pseudo images by
cylindrical projection (point-to-image structure alignment) and perform
adaptive global feature fusion between point features with local fused
features. Our method achieves state-of-the-art performance on KITTI odometry
and FlyingThings3D scene flow datasets compared to both single-modal and
multi-modal methods. Codes will be released later.

通过局部到全局的融合网络与双向结构对齐，本研究解决了视觉 - LiDAR 融合中由于两种模态之间内在数据结构不一致性而带来的挑战，取得了在 KITTI 里程计和 FlyingThings3D 场景流数据集上超过单模态和多模态方法的最新成果。