Multimodal medical imaging plays a pivotal role in clinical diagnosis and
research, as it combines information from various imaging modalities to provide
a more comprehensive understanding of the underlying pathology. Recently, deep
learning-based multimodal fusion techniques have emerged as powerful tools for
improving medical image classification. This review offers a thorough analysis
of the developments in deep learning-based multimodal fusion for medical
classification tasks. We explore the complementary relationships among
prevalent clinical modalities and outline three main fusion schemes for
multimodal classification networks: input fusion, intermediate fusion
(encompassing single-level fusion, hierarchical fusion, and attention-based
fusion), and output fusion. By evaluating the performance of these fusion
techniques, we provide insight into the suitability of different network
architectures for various multimodal fusion scenarios and application domains.
Furthermore, we delve into challenges related to network architecture
selection, handling incomplete multimodal data management, and the potential
limitations of multimodal fusion. Finally, we spotlight the promising future of
Transformer-based multimodal fusion techniques and give recommendations for
future research in this rapidly evolving field.

多模医学成像在临床诊断和研究中起着关键作用，深度学习多模态融合技术在提高医学图像分类方面显示出强大的工具。本综述对基于深度学习的医学分类任务中的多模态融合的发展进行了彻底分析，包括不同融合方案和网络架构的性能评估，讨论了相关挑战和未来研究方向。

基于深度学习的多模态医学图像分类信息融合技术综述

A review of deep learning-based information fusion techniques for  multimodal medical image classification

There are increasing concerns about malicious attacks on autonomous vehicles.
In particular, inaudible voice command attacks pose a significant threat as
voice commands become available in autonomous driving systems. How to
empirically defend against these inaudible attacks remains an open question.
Previous research investigates utilizing deep learning-based multimodal fusion
for defense, without considering the model uncertainty in trustworthiness. As
deep learning has been applied to increasingly sensitive tasks, uncertainty
measurement is crucial in helping improve model robustness, especially in
mission-critical scenarios. In this paper, we propose the Multimodal Fusion
Framework (MFF) as an intelligent security system to defend against inaudible
voice command attacks. MFF fuses heterogeneous audio-vision modalities using
VGG family neural networks and achieves the detection accuracy of 92.25% in the
comparative fusion method empirical study. Additionally, extensive experiments
on audio-vision tasks reveal the model's uncertainty. Using Expected
Calibration Errors, we measure calibration errors and Monte-Carlo Dropout to
estimate the predictive distribution for the proposed models. Our findings show
empirically to train robust multimodal models, improve standard accuracy and
provide a further step toward interpretability. Finally, we discuss the pros
and cons of our approach and its applicability for Advanced Driver Assistance
Systems.

本文提出了一种多模态融合 (Multimodal Fusion Framework) 安全系统，利用 VGG 系列神经网络融合异构音频 - 视觉模态，检测准确率可达 92.25%, 并测量了模型不确定性，结果表明我们的方法可帮助提高模型鲁棒性和准确性，并为理解提供了更深入的步骤，适用于高级驾驶辅助系统。