The integration of multimodal Electronic Health Records (EHR) data has significantly improved clinical predictive capabilities. Leveraging clinical notes and multivariate time-series EHR, existing models often lack the medical context relevent to clinical tasks, prompting the incorporation of external knowledge, particularly from the knowledge graph (KG). Previous approaches with KG knowledge have primarily focused on structured knowledge extraction, neglecting unstructured data modalities and semantic high dimensional medical knowledge. In response, we propose REALM, a Retrieval-Augmented Generation (RAG) driven framework to enhance multimodal EHR representations that address these limitations. Firstly, we apply Large Language Model (LLM) to encode long context clinical notes and GRU model to encode time-series EHR data. Secondly, we prompt LLM to extract task-relevant medical entities and match entities in professionally labeled external knowledge graph (PrimeKG) with corresponding medical knowledge. By matching and aligning with clinical standards, our framework eliminates hallucinations and ensures consistency. Lastly, we propose an adaptive multimodal fusion network to integrate extracted knowledge with multimodal EHR data. Our extensive experiments on MIMIC-III mortality and readmission tasks showcase the superior performance of our REALM framework over baselines, emphasizing the effectiveness of each module. REALM framework contributes to refining the use of multimodal EHR data in healthcare and bridging the gap with nuanced medical context essential for informed clinical predictions.

提出了一种名为REALM的框架，通过整合多模态电子健康记录（EHR）数据和知识图谱 (KG)，利用大型语言模型 (LLM) 编码临床笔记，并使用 GRU 模型编码时间序列 EHR 数据，提取任务相关的医疗实体，并将这些实体与专业标记的外部知识图谱 (PrimeKG) 进行匹配，通过适应性多模态融合网络将提取的知识与多模态 EHR 数据进行整合，最后通过对MIMIC-III的死亡率和再入院任务的大量实验，证明了REALM框架相比基线模型具有更优异的性能，强调了每个模块的有效性，从而改进了在医疗保健中利用多模态 EHR 数据并弥合了为明智临床预测所必需的细致医疗背景的差距。

REALM: 通过大型语言模型驱动的RAG的多模态电子健康记录分析的增强