Electronic Health Records (EHRs), which contain patients' medical histories
in various multi-modal formats, often overlook the potential for joint
reasoning across imaging and table modalities underexplored in current EHR
Question Answering (QA) systems. In this paper, we introduce EHRXQA, a novel
multi-modal question answering dataset combining structured EHRs and chest
X-ray images. To develop our dataset, we first construct two uni-modal
resources: 1) The MIMIC- CXR-VQA dataset, our newly created medical visual
question answering (VQA) benchmark, specifically designed to augment the
imaging modality in EHR QA, and 2) EHRSQL (MIMIC-IV), a refashioned version of
a previously established table-based EHR QA dataset. By integrating these two
uni-modal resources, we successfully construct a multi-modal EHR QA dataset
that necessitates both uni-modal and cross-modal reasoning. To address the
unique challenges of multi-modal questions within EHRs, we propose a
NeuralSQL-based strategy equipped with an external VQA API. This pioneering
endeavor enhances engagement with multi-modal EHR sources and we believe that
our dataset can catalyze advances in real-world medical scenarios such as
clinical decision-making and research. EHRXQA is available at
this https URL

通过整合医学影像和表格模式，本研究提出了一个新颖的多模式电子健康问答数据集（EHRXQA），以增强当前电子健康问答系统中的联合推理潜力，并改进真实医学场景中的临床决策和研究。