Multi-sensor fusion is essential for accurate 3D object detection in
self-driving systems. Camera and LiDAR are the most commonly used sensors, and
usually, their fusion happens at the early or late stages of 3D detectors with
the help of regions of interest (RoIs). On the other hand, fusion at the
intermediate level is more adaptive because it does not need RoIs from
modalities but is complex as the features of both modalities are presented from
different points of view. In this paper, we propose a new intermediate-level
multi-modal fusion (mmFUSION) approach to overcome these challenges. First, the
mmFUSION uses separate encoders for each modality to compute features at a
desired lower space volume. Second, these features are fused through
cross-modality and multi-modality attention mechanisms proposed in mmFUSION.
The mmFUSION framework preserves multi-modal information and learns to
complement modalities' deficiencies through attention weights. The strong
multi-modal features from the mmFUSION framework are fed to a simple 3D
detection head for 3D predictions. We evaluate mmFUSION on the KITTI and
NuScenes dataset where it performs better than available early, intermediate,
late, and even two-stage based fusion schemes. The code with the mmdetection3D
project plugin will be publicly available soon.

提出一种新的中级多模态融合（mmFUSION）方法，通过分别计算每种模态的特征并通过交叉模态和多模态注意机制进行融合，实现了精确的自动驾驶系统中的三维目标检测。