Multimodal deep sensor fusion has the potential to enable autonomous vehicles
to visually understand their surrounding environments in all weather
conditions. However, existing deep sensor fusion methods usually employ
convoluted architectures with intermingled multimodal features, requiring large
coregistered multimodal datasets for training. In this work, we present an
efficient and modular RGB-X fusion network that can leverage and fuse
pretrained single-modal models via scene-specific fusion modules, thereby
enabling joint input-adaptive network architectures to be created using small,
coregistered multimodal datasets. Our experiments demonstrate the superiority
of our method compared to existing works on RGB-thermal and RGB-gated datasets,
performing fusion using only a small amount of additional parameters. Our code
is available at this https URL

多模式深度传感器融合为自动驾驶车辆在各种天气条件下实现视觉理解提供了潜力，本研究提出了一种高效、模块化的 RGB-X 融合网络，通过场景特定的融合模块，实现了利用预训练的单模态模型进行融合，从而使用小型、核心配准的多模态数据集创建联合输入自适应的网络架构，在 RGB - 热成像和 RGB - 门控数据集上表现出了优越性，仅使用少量额外的参数进行融合。