The integration of multimodal Electronic Health Records (EHR) data has
notably advanced clinical predictive capabilities. However, current models that
utilize clinical notes and multivariate time-series EHR data often lack the
necessary medical context for precise clinical tasks. Previous methods using
knowledge graphs (KGs) primarily focus on structured knowledge extraction. To
address this, we propose EMERGE, a Retrieval-Augmented Generation (RAG) driven
framework aimed at enhancing multimodal EHR predictive modeling. Our approach
extracts entities from both time-series data and clinical notes by prompting
Large Language Models (LLMs) and aligns them with professional PrimeKG to
ensure consistency. Beyond triplet relationships, we include entities'
definitions and descriptions to provide richer semantics. The extracted
knowledge is then used to generate task-relevant summaries of patients' health
statuses. These summaries are fused with other modalities utilizing an adaptive
multimodal fusion network with cross-attention. Extensive experiments on the
MIMIC-III and MIMIC-IV datasets for in-hospital mortality and 30-day
readmission tasks demonstrate the superior performance of the EMERGE framework
compared to baseline models. Comprehensive ablation studies and analyses
underscore the efficacy of each designed module and the framework's robustness
to data sparsity. EMERGE significantly enhances the use of multimodal EHR data
in healthcare, bridging the gap with nuanced medical contexts crucial for
informed clinical predictions.

利用 EMERGE 框架、Retrieval-Augmented Generation (RAG) 驅動，提取多模態電子健康紀錄 (EHR) 中的實體，並結合專業 PrimeKG 確保一致性，以加強多模態 EHR 預測建模並生成患者健康狀態相關摘要，通過交叉注意的自適應多模態融合網絡與其他模態進行融合，並在 MIMIC-III 和 MIMIC-IV 資料集上的實驗中證明 EMERGE 框架相對於基線模型的優越性，徹底研究和分析驗證了每個設計模組的有效性，以及該框架對於數據稀疏性的魯棒性。EMERGE 顯著提高了多模態 EHR 數據在醫療保健中的應用，彌補了對於知情臨床預測至關重要的細緻醫學背景的差距。

融合 RAG 以提升多模态电子健康记录预测建模的 EMERGE

EMERGE: Integrating RAG for Improved Multimodal EHR Predictive Modeling

The integration of multimodal Electronic Health Records (EHR) data has
significantly improved clinical predictive capabilities. Leveraging clinical
notes and multivariate time-series EHR, existing models often lack the medical
context relevent to clinical tasks, prompting the incorporation of external
knowledge, particularly from the knowledge graph (KG). Previous approaches with
KG knowledge have primarily focused on structured knowledge extraction,
neglecting unstructured data modalities and semantic high dimensional medical
knowledge. In response, we propose REALM, a Retrieval-Augmented Generation
(RAG) driven framework to enhance multimodal EHR representations that address
these limitations. Firstly, we apply Large Language Model (LLM) to encode long
context clinical notes and GRU model to encode time-series EHR data. Secondly,
we prompt LLM to extract task-relevant medical entities and match entities in
professionally labeled external knowledge graph (PrimeKG) with corresponding
medical knowledge. By matching and aligning with clinical standards, our
framework eliminates hallucinations and ensures consistency. Lastly, we propose
an adaptive multimodal fusion network to integrate extracted knowledge with
multimodal EHR data. Our extensive experiments on MIMIC-III mortality and
readmission tasks showcase the superior performance of our REALM framework over
baselines, emphasizing the effectiveness of each module. REALM framework
contributes to refining the use of multimodal EHR data in healthcare and
bridging the gap with nuanced medical context essential for informed clinical
predictions.

提出了一种名为 REALM 的框架，通过整合多模态电子健康记录（EHR）数据和知识图谱 (KG)，利用大型语言模型 (LLM) 编码临床笔记，并使用 GRU 模型编码时间序列 EHR 数据，提取任务相关的医疗实体，并将这些实体与专业标记的外部知识图谱 (PrimeKG) 进行匹配，通过适应性多模态融合网络将提取的知识与多模态 EHR 数据进行整合，最后通过对 MIMIC-III 的死亡率和再入院任务的大量实验，证明了 REALM 框架相比基线模型具有更优异的性能，强调了每个模块的有效性，从而改进了在医疗保健中利用多模态 EHR 数据并弥合了为明智临床预测所必需的细致医疗背景的差距。

REALM: 通过大型语言模型驱动的 RAG 的多模态电子健康记录分析的增强

REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records  Analysis via Large Language Models

With the emergence of multimodal electronic health records, the evidence for
an outcome may be captured across multiple modalities ranging from clinical to
imaging and genomic data. Predicting outcomes effectively requires fusion
frameworks capable of modeling fine-grained and multi-faceted complex
interactions between modality features within and across patients. We develop
an innovative fusion approach called MaxCorr MGNN that models non-linear
modality correlations within and across patients through
Hirschfeld-Gebelein-Renyi maximal correlation (MaxCorr) embeddings, resulting
in a multi-layered graph that preserves the identities of the modalities and
patients. We then design, for the first time, a generalized multi-layered graph
neural network (MGNN) for task-informed reasoning in multi-layered graphs, that
learns the parameters defining patient-modality graph connectivity and message
passing in an end-to-end fashion. We evaluate our model an outcome prediction
task on a Tuberculosis (TB) dataset consistently outperforming several
state-of-the-art neural, graph-based and traditional fusion techniques.

本文提出了一种称为 MaxCorr MGNN 的创新融合方法，旨在通过 Hirschfeld-Gebelein-Renyi 最大相关性嵌入来建模跨患者内部和之间的非线性模式相关性，从而得出一个多层图模型，从而有效地预测结局。通过设计一种广义的多层图神经网络（MGNN），实现患者模式图的功能驱动推理和端到端的信息传递。我们在肺结核数据集上进行了实验，并始终优于几种最先进的神经网络，基于图的，传统的融合技术。