The integration of multimodal Electronic Health Records (EHR) data has
notably advanced clinical predictive capabilities. However, current models that
utilize clinical notes and multivariate time-series EHR data often lack the
necessary medical context for precise clinical tasks. Previous methods using
knowledge graphs (KGs) primarily focus on structured knowledge extraction. To
address this, we propose EMERGE, a Retrieval-Augmented Generation (RAG) driven
framework aimed at enhancing multimodal EHR predictive modeling. Our approach
extracts entities from both time-series data and clinical notes by prompting
Large Language Models (LLMs) and aligns them with professional PrimeKG to
ensure consistency. Beyond triplet relationships, we include entities'
definitions and descriptions to provide richer semantics. The extracted
knowledge is then used to generate task-relevant summaries of patients' health
statuses. These summaries are fused with other modalities utilizing an adaptive
multimodal fusion network with cross-attention. Extensive experiments on the
MIMIC-III and MIMIC-IV datasets for in-hospital mortality and 30-day
readmission tasks demonstrate the superior performance of the EMERGE framework
compared to baseline models. Comprehensive ablation studies and analyses
underscore the efficacy of each designed module and the framework's robustness
to data sparsity. EMERGE significantly enhances the use of multimodal EHR data
in healthcare, bridging the gap with nuanced medical contexts crucial for
informed clinical predictions.

利用 EMERGE 框架、Retrieval-Augmented Generation (RAG) 驅動，提取多模態電子健康紀錄 (EHR) 中的實體，並結合專業 PrimeKG 確保一致性，以加強多模態 EHR 預測建模並生成患者健康狀態相關摘要，通過交叉注意的自適應多模態融合網絡與其他模態進行融合，並在 MIMIC-III 和 MIMIC-IV 資料集上的實驗中證明 EMERGE 框架相對於基線模型的優越性，徹底研究和分析驗證了每個設計模組的有效性，以及該框架對於數據稀疏性的魯棒性。EMERGE 顯著提高了多模態 EHR 數據在醫療保健中的應用，彌補了對於知情臨床預測至關重要的細緻醫學背景的差距。