In image fusion, images obtained from different sensors are fused to generate
a single image with enhanced information. In recent years, state-of-the-art
methods have adopted Convolution Neural Networks (CNNs) to encode meaningful
features for image fusion. Specifically, CNN-based methods perform image fusion
by fusing local features. However, they do not consider long-range dependencies
that are present in the image. Transformer-based models are designed to
overcome this by modeling the long-range dependencies with the help of
self-attention mechanism. This motivates us to propose a novel Image Fusion
Transformer (IFT) where we develop a transformer-based multi-scale fusion
strategy that attends to both local and long-range information (or global
context). The proposed method follows a two-stage training approach. In the
first stage, we train an auto-encoder to extract deep features at multiple
scales. In the second stage, multi-scale features are fused using a
Spatio-Transformer (ST) fusion strategy. The ST fusion blocks are comprised of
a CNN and a transformer branch which capture local and long-range features,
respectively. Extensive experiments on multiple benchmark datasets show that
the proposed method performs better than many competitive fusion algorithms.
Furthermore, we show the effectiveness of the proposed ST fusion strategy with
an ablation analysis. The source code is available at:
this https URL

本研究提出一种新的基于 Transformer 的图像融合方法，采用多尺度融合策略同时关注局部和全局信息，使用卷积神经网络与 Transformer 分支捕捉局部和长程特征，经对比实验表明，该方法优于当下多种融合算法。