Advanced image fusion methods are devoted to generating the fusion results by
aggregating the complementary information conveyed by the source images.
However, the difference in the source-specific manifestation of the imaged
scene content makes it difficult to design a robust and controllable fusion
process. We argue that this issue can be alleviated with the help of
higher-level semantics, conveyed by the text modality, which should enable us
to generate fused images for different purposes, such as visualisation and
downstream tasks, in a controllable way. This is achieved by exploiting a
vision-and-language model to build a coarse-to-fine association mechanism
between the text and image signals. With the guidance of the association maps,
an affine fusion unit is embedded in the transformer network to fuse the text
and vision modalities at the feature level. As another ingredient of this work,
we propose the use of textual attention to adapt image quality assessment to
the fusion task. To facilitate the implementation of the proposed text-guided
fusion paradigm, and its adoption by the wider research community, we release a
text-annotated image fusion dataset IVT. Extensive experiments demonstrate that
our approach (TextFusion) consistently outperforms traditional appearance-based
fusion methods. Our code and dataset will be publicly available on the project
homepage.

利用高级语义，通过视觉 - 语言模型建立文本与图像信号之间的粗粒度关联机制，并在变换器网络中嵌入仿射融合单元以在特征级别上融合文本和图像模态，提出了一种以文本为导向的融合方法，并释放了一个文本注释的图像融合数据集 IVT 来促进其实施和在研究社区的采用。大量实验证明我们的方法（TextFusion）在传统的基于外观的融合方法上始终表现优异。