In this paper, we propose a novel deep convolutional neural network to solve
the general multi-modal image restoration (MIR) and multi-modal image fusion
(MIF) problems. Different from other methods based on deep learning, our
network architecture is designed by drawing inspirations from a new proposed
multi-modal convolutional sparse coding (MCSC) model. The key feature of the
proposed network is that it can automatically split the common information
shared among different modalities, from the unique information that belongs to
each single modality, and is therefore denoted with CU-Net, i.e., Common and
Unique information splitting network. Specifically, the CU-Net is composed of
three modules, i.e., the unique feature extraction module (UFEM), common
feature preservation module (CFPM), and image reconstruction module (IRM). The
architecture of each module is derived from the corresponding part in the MCSC
model, which consists of several learned convolutional sparse coding (LCSC)
blocks. Extensive numerical results verify the effectiveness of our method on a
variety of MIR and MIF tasks, including RGB guided depth image
super-resolution, flash guided non-flash image denoising, multi-focus and
multi-exposure image fusion.

本论文提出了一种新颖的深度卷积神经网络，用于解决多模态图像恢复和融合问题，其架构从多模态卷积稀疏编码模型中得到启发，具有自动分离共享于不同模态之间的公共信息和属于单一模态的独特信息的特点，命名为 CU-Net, 由 3 个模块组成，可以在各种 MIR 和 MIF 任务中发挥作用。