With the rapid progression of deep learning technologies, multi-modality
image fusion has become increasingly prevalent in object detection tasks.
Despite its popularity, the inherent disparities in how different sources
depict scene content make fusion a challenging problem. Current fusion
methodologies identify shared characteristics between the two modalities and
integrate them within this shared domain using either iterative optimization or
deep learning architectures, which often neglect the intricate semantic
relationships between modalities, resulting in a superficial understanding of
inter-modal connections and, consequently, suboptimal fusion outcomes. To
address this, we introduce a text-guided multi-modality image fusion method
that leverages the high-level semantics from textual descriptions to integrate
semantics from infrared and visible images. This method capitalizes on the
complementary characteristics of diverse modalities, bolstering both the
accuracy and robustness of object detection. The codebook is utilized to
enhance a streamlined and concise depiction of the fused intra- and
inter-domain dynamics, fine-tuned for optimal performance in detection tasks.
We present a bilevel optimization strategy that establishes a nexus between the
joint problem of fusion and detection, optimizing both processes concurrently.
Furthermore, we introduce the first dataset of paired infrared and visible
images accompanied by text prompts, paving the way for future research.
Extensive experiments on several datasets demonstrate that our method not only
produces visually superior fusion results but also achieves a higher detection
mAP over existing methods, achieving state-of-the-art results.

本研究提出了一种基于文本引导的多模态图像融合方法，利用文本描述的高级语义结合红外和可见光图像的语义信息，为目标检测任务提供了更准确和鲁棒的结果。通过使用代码本来增强对融合动态的简明直观表达，并通过双层优化策略同时优化融合和检测问题，本研究取得了与现有方法相比更高的检测平均精度和视觉上优越的融合结果。

从文本到像素：一种针对红外和可见光图像融合的上下文感知语义协同解决方案

From Text to Pixels: A Context-Aware Semantic Synergy Solution for  Infrared and Visible Image Fusion

Advanced image fusion methods are devoted to generating the fusion results by
aggregating the complementary information conveyed by the source images.
However, the difference in the source-specific manifestation of the imaged
scene content makes it difficult to design a robust and controllable fusion
process. We argue that this issue can be alleviated with the help of
higher-level semantics, conveyed by the text modality, which should enable us
to generate fused images for different purposes, such as visualisation and
downstream tasks, in a controllable way. This is achieved by exploiting a
vision-and-language model to build a coarse-to-fine association mechanism
between the text and image signals. With the guidance of the association maps,
an affine fusion unit is embedded in the transformer network to fuse the text
and vision modalities at the feature level. As another ingredient of this work,
we propose the use of textual attention to adapt image quality assessment to
the fusion task. To facilitate the implementation of the proposed text-guided
fusion paradigm, and its adoption by the wider research community, we release a
text-annotated image fusion dataset IVT. Extensive experiments demonstrate that
our approach (TextFusion) consistently outperforms traditional appearance-based
fusion methods. Our code and dataset will be publicly available on the project
homepage.

利用高级语义，通过视觉 - 语言模型建立文本与图像信号之间的粗粒度关联机制，并在变换器网络中嵌入仿射融合单元以在特征级别上融合文本和图像模态，提出了一种以文本为导向的融合方法，并释放了一个文本注释的图像融合数据集 IVT 来促进其实施和在研究社区的采用。大量实验证明我们的方法（TextFusion）在传统的基于外观的融合方法上始终表现优异。