Image analysis using more than one modality (i.e. multi-modal) has been
increasingly applied in the field of biomedical imaging. One of the challenges
in performing the multimodal analysis is that there exist multiple schemes for
fusing the information from different modalities, where such schemes are
application-dependent and lack a unified framework to guide their designs. In
this work we firstly propose a conceptual architecture for the image fusion
schemes in supervised biomedical image analysis: fusing at the feature level,
fusing at the classifier level, and fusing at the decision-making level.
Further, motivated by the recent success in applying deep learning for natural
image analysis, we implement the three image fusion schemes above based on the
Convolutional Neural Network (CNN) with varied structures, and combined into a
single framework. The proposed image segmentation framework is capable of
analyzing the multi-modality images using different fusing schemes
simultaneously. The framework is applied to detect the presence of soft tissue
sarcoma from the combination of Magnetic Resonance Imaging (MRI), Computed
Tomography (CT) and Positron Emission Tomography (PET) images. It is found from
the results that while all the fusion schemes outperform the single-modality
schemes, fusing at the feature level can generally achieve the best performance
in terms of both accuracy and computational cost, but also suffers from the
decreased robustness in the presence of large errors in any image modalities.

本文提出了在监督医学图像分析中进行图像融合方案的概念架构，实现了基于卷积神经网络的三种图像融合方案，并将其结合成单个框架，用于同时分析多模式图像，检测软组织肉瘤的存在，并发现在特征级别融合的情况下，性能最好，但当任何图像模式存在大误差时，其鲁棒性会降低。