Multispectral pedestrian detection has been shown to be effective in
improving performance within complex illumination scenarios. However, prevalent
double-stream networks in multispectral detection employ two separate feature
extraction branches for multi-modal data, leading to nearly double the
inference time compared to single-stream networks utilizing only one feature
extraction branch. This increased inference time has hindered the widespread
employment of multispectral pedestrian detection in embedded devices for
autonomous systems. To address this limitation, various knowledge distillation
methods have been proposed. However, traditional distillation methods focus
only on the fusion features and ignore the large amount of information in the
original multi-modal features, thereby restricting the student network's
performance. To tackle the challenge, we introduce the Adaptive Modal Fusion
Distillation (AMFD) framework, which can fully utilize the original modal
features of the teacher network. Specifically, a Modal Extraction Alignment
(MEA) module is utilized to derive learning weights for student networks,
integrating focal and global attention mechanisms. This methodology enables the
student network to acquire optimal fusion strategies independent from that of
teacher network without necessitating an additional feature fusion module.
Furthermore, we present the SMOD dataset, a well-aligned challenging
multispectral dataset for detection. Extensive experiments on the challenging
KAIST, LLVIP and SMOD datasets are conducted to validate the effectiveness of
AMFD. The results demonstrate that our method outperforms existing
state-of-the-art methods in both reducing log-average Miss Rate and improving
mean Average Precision. The code is available at
this https URL

多光谱行人检测中，采用自适应模态融合蒸馏（AMFD）框架，可有效利用原始模态特征，通过模态提取对齐模块提供学生网络的学习权重，进而独立于教师网络获得最佳融合策略，并在 KAIST、LLVIP 和 SMOD 数据集上实验证明了该方法在降低漏检率和提高平均精度方面优于现有最先进方法。