Autonomous vehicles (AVs) must accurately detect objects from both common and
rare classes for safe navigation, motivating the problem of Long-Tailed 3D
Object Detection (LT3D). Contemporary LiDAR-based 3D detectors perform poorly
on rare classes (e.g., CenterPoint only achieves 5.1 AP on stroller) as it is
difficult to recognize objects from sparse LiDAR points alone. RGB images
provide visual evidence to help resolve such ambiguities, motivating the study
of RGB-LiDAR fusion. In this paper, we delve into a simple late-fusion
framework that ensembles independently trained RGB and LiDAR detectors. Unlike
recent end-to-end methods which require paired multi-modal training data, our
late-fusion approach can easily leverage large-scale uni-modal datasets,
significantly improving rare class detection.In particular, we examine three
critical components in this late-fusion framework from first principles,
including whether to train 2D or 3D RGB detectors, whether to match RGB and
LiDAR detections in 3D or the projected 2D image plane, and how to fuse matched
detections.Extensive experiments reveal that 2D RGB detectors achieve better
recognition accuracy than 3D RGB detectors, matching on the 2D image plane
mitigates depth estimation errors, and fusing scores probabilistically with
calibration leads to state-of-the-art LT3D performance. Our late-fusion
approach achieves 51.4 mAP on the established nuScenes LT3D benchmark,
improving over prior work by 5.9 mAP.

自动驾驶车辆要准确检测常见和罕见类别的物体以确保安全导航，因此出现了长尾三维物体检测（LT3D）的问题。本文研究了 RGB-LiDAR 融合的简单的后期融合框架，通过集成独立训练的 RGB 和 LiDAR 检测器，利用大规模单模数据集显著提高了罕见类别的检测性能，在几个关键组件方面进行了探讨，包括训练二维或三维 RGB 检测器、在三维空间中匹配 RGB 和 LiDAR 检测结果或在二维图像平面中匹配、以及如何概率地融合匹配的检测结果。大量实验结果显示，二维 RGB 检测器的识别准确性优于三维 RGB 检测器，基于二维图像平面的匹配可以减轻深度估计误差，使用概率校准的方法将分数融合可以达到最先进的 LT3D 性能。我们的后期融合方法在 nuScenes LT3D 基准测试中取得了 51.4 mAP 的成绩，相比之前的工作提高了 5.9 mAP。