RGB-Thermal (RGB-T) pedestrian detection aims to locate the pedestrians in
RGB-T image pairs to exploit the complementation between the two modalities for
improving detection robustness in extreme conditions. Most existing algorithms
assume that the RGB-T image pairs are well registered, while in the real world
they are not aligned ideally due to parallax or different field-of-view of the
cameras. The pedestrians in misaligned image pairs may locate at different
positions in two images, which results in two challenges: 1) how to achieve
inter-modality complementation using spatially misaligned RGB-T pedestrian
patches, and 2) how to recognize the unpaired pedestrians at the boundary. To
deal with these issues, we propose a new paradigm for unregistered RGB-T
pedestrian detection, which predicts two separate pedestrian locations in the
RGB and thermal images, respectively. Specifically, we propose a cross-modality
proposal-guided feature mining (CPFM) mechanism to extract the two precise
fusion features for representing the pedestrian in the two modalities, even if
the RGB-T image pair is unaligned. It enables us to effectively exploit the
complementation between the two modalities. With the CPFM mechanism, we build a
two-stream dense detector; it predicts the two pedestrian locations in the two
modalities based on the corresponding fusion feature mined by the CPFM
mechanism. Besides, we design a data augmentation method, named Homography, to
simulate the discrepancy in scales and views between images. We also
investigate two non-maximum suppression (NMS) methods for post-processing.
Favorable experimental results demonstrate the effectiveness and robustness of
our method in dealing with unregistered pedestrians with different shifts.

通过提出跨模态提案引导特征挖掘机制和双流密集检测器，本研究解决了 RGB-T 行人检测中由于未对齐图像对导致的跨模态互补和未对齐行人的识别等问题，并通过数据增强方法和非极大值抑制方法进一步提高了方法的有效性和鲁棒性。