Multi-modal image fusion (MMIF) integrates valuable information from
different modality images into a fused one. However, the fusion of multiple
visible images with different focal regions and infrared images is a
unprecedented challenge in real MMIF applications. This is because of the
limited depth of the focus of visible optical lenses, which impedes the
simultaneous capture of the focal information within the same scene. To address
this issue, in this paper, we propose a MMIF framework for joint focused
integration and modalities information extraction. Specifically, a
semi-sparsity-based smoothing filter is introduced to decompose the images into
structure and texture components. Subsequently, a novel multi-scale operator is
proposed to fuse the texture components, capable of detecting significant
information by considering the pixel focus attributes and relevant data from
various modal images. Additionally, to achieve an effective capture of scene
luminance and reasonable contrast maintenance, we consider the distribution of
energy information in the structural components in terms of multi-directional
frequency variance and information entropy. Extensive experiments on existing
MMIF datasets, as well as the object detection and depth estimation tasks,
consistently demonstrate that the proposed algorithm can surpass the
state-of-the-art methods in visual perception and quantitative evaluation. The
code is available at this https URL

在本文中，我们提出了一种多模态图像融合（MMIF）框架，旨在解决多个可见光图像与红外图像的焦点区域不同的融合问题。通过引入半稀疏平滑滤波器对图像进行分解为结构和纹理成分，并提出了一种新的多尺度操作器对纹理成分进行融合。同时，我们考虑结构成分中能量信息的分布，以实现场景亮度的有效捕捉和合理的对比度维持。实验结果表明，该算法在视觉感知和数量化评估方面均优于现有的方法。