Multimodal visual information fusion aims to integrate the multi-sensor data
into a single image which contains more complementary information and less
redundant features. However the complementary information is hard to extract,
especially for infrared and visible images which contain big similarity gap
between these two modalities. The common cross attention modules only consider
the correlation, on the contrary, image fusion tasks need focus on
complementarity (uncorrelation). Hence, in this paper, a novel cross attention
mechanism (CAM) is proposed to enhance the complementary information.
Furthermore, a two-stage training strategy based fusion scheme is presented to
generate the fused images. For the first stage, two auto-encoder networks with
same architecture are trained for each modality. Then, with the fixed encoders,
the CAM and a decoder are trained in the second stage. With the trained CAM,
features extracted from two modalities are integrated into one fused feature in
which the complementary information is enhanced and the redundant features are
reduced. Finally, the fused image can be generated by the trained decoder. The
experimental results illustrate that our proposed fusion method obtains the
SOTA fusion performance compared with the existing fusion networks. The codes
are available at this https URL

提出了一种新颖的跨模态注意机制（CAM）来增强互补信息，并提出了基于两阶段训练策略的融合方案，实验结果表明我们提出的融合方法在融合网络中获得了 SOTA 级别的性能。

CrossFuse：一种基于交叉注意力机制的红外和可见光图像融合新方法

CrossFuse: A Novel Cross Attention Mechanism based Infrared and Visible  Image Fusion Approach

The recent surge of interest surrounding Multimodal Neural Networks (MM-NN)
is attributed to their ability to effectively process and integrate information
from diverse data sources. In MM-NN, features are extracted and fused from
multiple modalities using adequate unimodal backbones and specific fusion
networks. Although this helps strengthen the multimodal information
representation, designing such networks is labor-intensive. It requires tuning
the architectural parameters of the unimodal backbones, choosing the fusing
point, and selecting the operations for fusion. Furthermore, multimodality AI
is emerging as a cutting-edge option in Internet of Things (IoT) systems where
inference latency and energy consumption are critical metrics in addition to
accuracy. In this paper, we propose Harmonic-NAS, a framework for the joint
optimization of unimodal backbones and multimodal fusion networks with hardware
awareness on resource-constrained devices. Harmonic-NAS involves a two-tier
optimization approach for the unimodal backbone architectures and fusion
strategy and operators. By incorporating the hardware dimension into the
optimization, evaluation results on various devices and multimodal datasets
have demonstrated the superiority of Harmonic-NAS over state-of-the-art
approaches achieving up to 10.9% accuracy improvement, 1.91x latency reduction,
and 2.14x energy efficiency gain.

提出了一种用于多模态融合网络的联合优化框架，通过嵌入硬件维度进行优化，结果表明在各种设备和多模态数据集上，Harmonic-NAS 相对于现有方法具有更高的准确率改进、更低的推理延迟和更高的能效提升。