Multi-modality image fusion is a technique used to combine information from
different sensors or modalities, allowing the fused image to retain
complementary features from each modality, such as functional highlights and
texture details. However, effectively training such fusion models is difficult
due to the lack of ground truth fusion data. To address this issue, we propose
the Equivariant Multi-Modality imAge fusion (EMMA) paradigm for end-to-end
self-supervised learning. Our approach is based on the prior knowledge that
natural images are equivariant to specific transformations. Thus, we introduce
a novel training framework that includes a fusion module and a learnable
pseudo-sensing module, which allow the network training to follow the
principles of physical sensing and imaging process, and meanwhile satisfy the
equivariant prior for natural images. Our extensive experiments demonstrate
that our method produces high-quality fusion results for both infrared-visible
and medical images, while facilitating downstream multi-modal segmentation and
detection tasks. The code will be released.

本文提出 Equivariant Multi-Modality imAge fusion (EMMA) paradigm 用于多模态图像融合，采用自我监督学习方法进行高质量的图像融合，并能够促进下游的多模态分割和检测任务。