Multimodal medical imaging plays a pivotal role in clinical diagnosis and
research, as it combines information from various imaging modalities to provide
a more comprehensive understanding of the underlying pathology. Recently, deep
learning-based multimodal fusion techniques have emerged as powerful tools for
improving medical image classification. This review offers a thorough analysis
of the developments in deep learning-based multimodal fusion for medical
classification tasks. We explore the complementary relationships among
prevalent clinical modalities and outline three main fusion schemes for
multimodal classification networks: input fusion, intermediate fusion
(encompassing single-level fusion, hierarchical fusion, and attention-based
fusion), and output fusion. By evaluating the performance of these fusion
techniques, we provide insight into the suitability of different network
architectures for various multimodal fusion scenarios and application domains.
Furthermore, we delve into challenges related to network architecture
selection, handling incomplete multimodal data management, and the potential
limitations of multimodal fusion. Finally, we spotlight the promising future of
Transformer-based multimodal fusion techniques and give recommendations for
future research in this rapidly evolving field.

多模医学成像在临床诊断和研究中起着关键作用，深度学习多模态融合技术在提高医学图像分类方面显示出强大的工具。本综述对基于深度学习的医学分类任务中的多模态融合的发展进行了彻底分析，包括不同融合方案和网络架构的性能评估，讨论了相关挑战和未来研究方向。

基于深度学习的多模态医学图像分类信息融合技术综述

A review of deep learning-based information fusion techniques for  multimodal medical image classification

Despite the routine use of electronic health record (EHR) data by
radiologists to contextualize clinical history and inform image interpretation,
the majority of deep learning architectures for medical imaging are unimodal,
i.e., they only learn features from pixel-level information. Recent research
revealing how race can be recovered from pixel data alone highlights the
potential for serious biases in models which fail to account for demographics
and other key patient attributes. Yet the lack of imaging datasets which
capture clinical context, inclusive of demographics and longitudinal medical
history, has left multimodal medical imaging underexplored. To better assess
these challenges, we present RadFusion, a multimodal, benchmark dataset of 1794
patients with corresponding EHR data and high-resolution computed tomography
(CT) scans labeled for pulmonary embolism. We evaluate several representative
multimodal fusion models and benchmark their fairness properties across
protected subgroups, e.g., gender, race/ethnicity, age. Our results suggest
that integrating imaging and EHR data can improve classification performance
and robustness without introducing large disparities in the true positive rate
between population groups.

通过整合医学影像和电子健康记录，可以提升肺栓塞分类性能和稳健性，并在不引入不必要误差的情况下，在受保护子群体之间全面评估公平性属性。