This study introduces EHRNoteQA, a novel patient-specific question answering
benchmark tailored for evaluating Large Language Models (LLMs) in clinical
environments. Based on MIMIC-IV Electronic Health Record (EHR), a team of three
medical professionals has curated the dataset comprising 962 unique questions,
each linked to a specific patient's EHR clinical notes. What makes EHRNoteQA
distinct from existing EHR-based benchmarks is as follows: Firstly, it is the
first dataset to adopt a multi-choice question answering format, a design
choice that effectively evaluates LLMs with reliable scores in the context of
automatic evaluation, compared to other formats. Secondly, it requires an
analysis of multiple clinical notes to answer a single question, reflecting the
complex nature of real-world clinical decision-making where clinicians review
extensive records of patient histories. Our comprehensive evaluation on various
large language models showed that their scores on EHRNoteQA correlate more
closely with their performance in addressing real-world medical questions
evaluated by clinicians than their scores from other LLM benchmarks. This
underscores the significance of EHRNoteQA in evaluating LLMs for medical
applications and highlights its crucial role in facilitating the integration of
LLMs into healthcare systems. The dataset will be made available to the public
under PhysioNet credential access, promoting further research in this vital
field.

这项研究介绍了 EHRNoteQA，这是一个为评估临床环境中大型语言模型（LLMs）而定制的创新患者特定问题回答基准。它是首个采用多项选择问题回答格式的数据集，通过分析多个临床记录来回答单个问题，与其他格式相比，在自动评估中有效地评估 LLMs 的可靠性得分。在各种大型语言模型上进行的综合评估表明，它们在 EHRNoteQA 上的得分与通过临床医生评估的真实医学问题的表现更密切相关，凸显了 EHRNoteQA 在评估医学应用的 LLMs 中的重要性，以及它在促进 LLMs 融入医疗系统中的关键作用。该数据集将在 PhysioNet 凭据访问下向公众提供，促进这个重要领域的进一步研究。