Multi-sensor perception is crucial to ensure the reliability and accuracy in
autonomous driving system, while multi-object tracking (MOT) improves that by
tracing sequential movement of dynamic objects. Most current approaches for
multi-sensor multi-object tracking are either lack of reliability by tightly
relying on a single input source (e.g., center camera), or not accurate enough
by fusing the results from multiple sensors in post processing without fully
exploiting the inherent information. In this study, we design a generic
sensor-agnostic multi-modality MOT framework (mmMOT), where each modality
(i.e., sensors) is capable of performing its role independently to preserve
reliability, and further improving its accuracy through a novel multi-modality
fusion module. Our mmMOT can be trained in an end-to-end manner, enables joint
optimization for the base feature extractor of each modality and an adjacency
estimator for cross modality. Our mmMOT also makes the first attempt to encode
deep representation of point cloud in data association process in MOT. We
conduct extensive experiments to evaluate the effectiveness of the proposed
framework on the challenging KITTI benchmark and report state-of-the-art
performance. Code and models are available at
this https URL

该研究提出了一个通用的、传感器无关的多模态多目标跟踪框架，通过在数据关联过程中编码点云的深度表示，使每种模态能够独立执行其职能以保证其可靠性，并通过新型的多模态融合模块进一步提高其精度。该框架在 KITTI 基准上的表现达到了最新水平。