3D LiDAR (light detection and ranging) semantic segmentation is important in
scene understanding for many applications, such as auto-driving and robotics.
For example, for autonomous cars equipped with RGB cameras and LiDAR, it is
crucial to fuse complementary information from different sensors for robust and
accurate segmentation. Existing fusion-based methods, however, may not achieve
promising performance due to the vast difference between the two modalities. In
this work, we investigate a collaborative fusion scheme called perception-aware
multi-sensor fusion (PMF) to exploit perceptual information from two
modalities, namely, appearance information from RGB images and spatio-depth
information from point clouds. To this end, we first project point clouds to
the camera coordinates to provide spatio-depth information for RGB images.
Then, we propose a two-stream network to extract features from the two
modalities, separately, and fuse the features by effective residual-based
fusion modules. Moreover, we propose additional perception-aware losses to
measure the perceptual difference between the two modalities. Extensive
experiments on two benchmark data sets show the superiority of our method. For
example, on nuScenes, our PMF outperforms the state-of-the-art method by 0.8 in
mIoU.

本文章提出了一种基于感知信息的多传感器融合策略（PMF），用于自动驾驶等应用中的 3D LiDAR 语义分割，通过两个模态的外观和空间深度信息进行感知信息融合并且提出了感知相关的损失函数用于衡量两种模式之间的差异性，实验结果证明了该方法的优越性。