Machine learning (ML) applications in medical artificial intelligence (AI)
systems have shifted from traditional and statistical methods to increasing
application of deep learning models and even more recently generative models.
Recent years have seen a rise in the discovery of widely-available deep
learning architectures that support multimodal data integration, particularly
with images. The incorporation of multiple modalities into these models is a
thriving research topic, presenting its own unique challenges. In this work, we
discuss five challenges to multimodal AI as it pertains to ML (representation,
fusion, alignment, translation, and co-learning) and survey recent approaches
to addressing these challenges in the context of medical image-based clinical
decision support models. We conclude with a discussion of the future of the
field, suggesting directions that should be elucidated further for successful
clinical models and their translation to the clinical setting.

机器学习在医疗人工智能系统中的应用，从传统和统计方法转向深度学习模型甚至最近的生成模型。近年来，发现了支持多模态数据集成的广泛可用的深度学习架构，尤其是针对图像的。将多种模态融入到这些模型中是一个蓬勃发展的研究课题，提出了独特的挑战。本文讨论了针对医学基于图像的临床决策支持模型的多模态人工智能的五大挑战（表示、融合、对齐、翻译和协同学习），并调查了近期解决这些挑战的方法。我们最后讨论了该领域的未来，并提出了在成功的临床模型及其在临床环境中的转化方面应进一步阐明的方向。