For 3D object detection, both camera and lidar have been demonstrated to be
useful sensory devices for providing complementary information about the same
scenery with data representations in different modalities, e.g., 2D RGB image
vs 3D point cloud. An effective representation learning and fusion of such
multi-modal sensor data is necessary and critical for better 3D object
detection performance. To solve the problem, in this paper, we will introduce a
novel vision transformer-based 3D object detection model, namely FusionViT.
Different from the existing 3D object detection approaches, FusionViT is a
pure-ViT based framework, which adopts a hierarchical architecture by extending
the transformer model to embed both images and point clouds for effective
representation learning. Such multi-modal data embedding representations will
be further fused together via a fusion vision transformer model prior to
feeding the learned features to the object detection head for both detection
and localization of the 3D objects in the input scenery. To demonstrate the
effectiveness of FusionViT, extensive experiments have been done on real-world
traffic object detection benchmark datasets KITTI and Waymo Open. Notably, our
FusionViT model can achieve state-of-the-art performance and outperforms not
only the existing baseline methods that merely rely on camera images or lidar
point clouds, but also the latest multi-modal image-point cloud deep fusion
approaches.

通过 FusionViT 模型，在 KITTI 和 Waymo Open 数据集上进行了大量实验，实现了最先进的性能，并且优于仅依赖于相机图像或 Lidar 点云以及最新的多模态图像 - 点云深度融合方法。