LiDAR and camera are two critical sensors for multi-modal 3D semantic
segmentation and are supposed to be fused efficiently and robustly to promise
safety in various real-world scenarios. However, existing multi-modal methods
face two key challenges: 1) difficulty with efficient deployment and real-time
execution; and 2) drastic performance degradation under weak calibration
between LiDAR and cameras. To address these challenges, we propose CPGNet-LCF,
a new multi-modal fusion framework extending the LiDAR-only CPGNet. CPGNet-LCF
solves the first challenge by inheriting the easy deployment and real-time
capabilities of CPGNet. For the second challenge, we introduce a novel weak
calibration knowledge distillation strategy during training to improve the
robustness against the weak calibration. CPGNet-LCF achieves state-of-the-art
performance on the nuScenes and SemanticKITTI benchmarks. Remarkably, it can be
easily deployed to run in 20ms per frame on a single Tesla V100 GPU using
TensorRT TF16 mode. Furthermore, we benchmark performance over four weak
calibration levels, demonstrating the robustness of our proposed approach.

CPGNet-LCF 是一个新的多模态融合框架，通过继承 CPGNet 的易于部署和实时能力来解决合并 LiDAR 和相机之间弱校准的挑战，在训练过程中引入一种新颖的弱校准知识蒸馏策略以提高其对弱校准的鲁棒性，实现了在 nuScenes 和 SemanticKITTI 基准测试上的最先进性能，并且可在单个 Tesla V100 GPU 上以 20ms 每帧的速度运行，进一步展示了我们提出的方法的鲁棒性。