LiDAR and camera are two modalities available for 3D semantic segmentation in
autonomous driving. The popular LiDAR-only methods severely suffer from
inferior segmentation on small and distant objects due to insufficient laser
points, while the robust multi-modal solution is under-explored, where we
investigate three crucial inherent difficulties: modality heterogeneity,
limited sensor field of view intersection, and multi-modal data augmentation.
We propose a multi-modal 3D semantic segmentation model (MSeg3D) with joint
intra-modal feature extraction and inter-modal feature fusion to mitigate the
modality heterogeneity. The multi-modal fusion in MSeg3D consists of
geometry-based feature fusion GF-Phase, cross-modal feature completion, and
semantic-based feature fusion SF-Phase on all visible points. The multi-modal
data augmentation is reinvigorated by applying asymmetric transformations on
LiDAR point cloud and multi-camera images individually, which benefits the
model training with diversified augmentation transformations. MSeg3D achieves
state-of-the-art results on nuScenes, Waymo, and SemanticKITTI datasets. Under
the malfunctioning multi-camera input and the multi-frame point clouds input,
MSeg3D still shows robustness and improves the LiDAR-only baseline. Our code is
publicly available at https://github.com/jialeli1/lidarseg3d.

利用多模态数据的几何和语义特征，通过三个步骤来完成多模态 3D 语义分割任务。经过我们的研究，MSeg3D 在 nuScenes、Waymo 和 SemanticKITTI 数据集上取得了最先进的结果。