Currently prevalent multimodal 3D detection methods are built upon
LiDAR-based detectors that usually use dense Bird's-Eye-View (BEV) feature
maps. However, the cost of such BEV feature maps is quadratic to the detection
range, making it not suitable for long-range detection. Fully sparse
architecture is gaining attention as they are highly efficient in long-range
perception. In this paper, we study how to effectively leverage image modality
in the emerging fully sparse architecture. Particularly, utilizing instance
queries, our framework integrates the well-studied 2D instance segmentation
into the LiDAR side, which is parallel to the 3D instance segmentation part in
the fully sparse detector. This design achieves a uniform query-based fusion
framework in both the 2D and 3D sides while maintaining the fully sparse
characteristic. Extensive experiments showcase state-of-the-art results on the
widely used nuScenes dataset and the long-range Argoverse 2 dataset. Notably,
the inference speed of the proposed method under the long-range LiDAR
perception setting is 2.7 $\times$ faster than that of other state-of-the-art
multimodal 3D detection methods. Code will be released at
https://github.com/BraveGroup/FullySparseFusion.

本文研究如何在全稀疏架构中有效利用图像模态，并利用实例查询将 2D 实例分割与 3D 实例分割部分并行集成到 LiDAR 侧，从而实现了统一的基于查询的 2D 和 3D 融合框架，同时保持全稀疏特性。广泛的实验证明了在广泛使用的 nuScenes 数据集和长距离 Argoverse 2 数据集上的最先进性能。值得注意的是，所提出的方法在长距离 LiDAR 感知设置下的推理速度比其他最先进的多模式 3D 检测方法快 2.7 倍。