Artificial intelligence (AI) systems hold great promise to improve healthcare
over the next decades. Specifically, AI systems leveraging multiple data
sources and input modalities are poised to become a viable method to deliver
more accurate results and deployable pipelines across a wide range of
applications. In this work, we propose and evaluate a unified Holistic AI in
Medicine (HAIM) framework to facilitate the generation and testing of AI
systems that leverage multimodal inputs. Our approach uses generalizable data
pre-processing and machine learning modeling stages that can be readily adapted
for research and deployment in healthcare environments. We evaluate our HAIM
framework by training and characterizing 14,324 independent models based on
HAIM-MIMIC-MM, a multimodal clinical database (N=34,537 samples) containing
7,279 unique hospitalizations and 6,485 patients, spanning all possible input
combinations of 4 data modalities (i.e., tabular, time-series, text, and
images), 11 unique data sources and 12 predictive tasks. We show that this
framework can consistently and robustly produce models that outperform similar
single-source approaches across various healthcare demonstrations (by 6-33%),
including 10 distinct chest pathology diagnoses, along with length-of-stay and
48-hour mortality predictions. We also quantify the contribution of each
modality and data source using Shapley values, which demonstrates the
heterogeneity in data modality importance and the necessity of multimodal
inputs across different healthcare-relevant tasks. The generalizable properties
and flexibility of our Holistic AI in Medicine (HAIM) framework could offer a
promising pathway for future multimodal predictive systems in clinical and
operational healthcare settings.

本研究提出并评估了一种综合人工智能在医学领域的框架（HAIM），以促进利用多模态输入实现人工智能系统的生成和测试，该框架可在医疗保健环境中进行研究和部署，并通过使用 Shapley 值量化每个模态和数据源的贡献，展示了多模态输入在不同医疗任务中的必要性。