In this work, we present PoIFusion, a simple yet effective multi-modal 3D
object detection framework to fuse the information of RGB images and LiDAR
point clouds at the point of interest (abbreviated as PoI). Technically, our
PoIFusion follows the paradigm of query-based object detection, formulating
object queries as dynamic 3D boxes. The PoIs are adaptively generated from each
query box on the fly, serving as the keypoints to represent a 3D object and
play the role of basic units in multi-modal fusion. Specifically, we project
PoIs into the view of each modality to sample the corresponding feature and
integrate the multi-modal features at each PoI through a dynamic fusion block.
Furthermore, the features of PoIs derived from the same query box are
aggregated together to update the query feature. Our approach prevents
information loss caused by view transformation and eliminates the
computation-intensive global attention, making the multi-modal 3D object
detector more applicable. We conducted extensive experiments on the nuScenes
dataset to evaluate our approach. Remarkably, our PoIFusion achieves 74.9\% NDS
and 73.4\% mAP, setting a state-of-the-art record on the multi-modal 3D object
detection benchmark. Codes will be made available via
https://djiajunustc.github.io/projects/poifusion.

我们提出了 PoIFusion，这是一个简单而有效的多模态 3D 物体检测框架，用于在兴趣点（简称 PoI）处融合 RGB 图像和 LiDAR 点云的信息。