Electronic health records (EHRs) contain vast amounts of complex data, but
harmonizing and processing this information remains a challenging and costly
task requiring significant clinical expertise. While large language models
(LLMs) have shown promise in various healthcare applications, their potential
for abstracting medical concepts from EHRs remains largely unexplored. We
introduce EHRmonize, a framework leveraging LLMs to abstract medical concepts
from EHR data. Our study uses medication data from two real-world EHR databases
to evaluate five LLMs on two free-text extraction and six binary classification
tasks across various prompting strategies. GPT-4o's with 10-shot prompting
achieved the highest performance in all tasks, accompanied by Claude-3.5-Sonnet
in a subset of tasks. GPT-4o achieved an accuracy of 97% in identifying generic
route names, 82% for generic drug names, and 100% in performing binary
classification of antibiotics. While EHRmonize significantly enhances
efficiency, reducing annotation time by an estimated 60%, we emphasize that
clinician oversight remains essential. Our framework, available as a Python
package, offers a promising tool to assist clinicians in EHR data abstraction,
potentially accelerating healthcare research and improving data harmonization
processes.

EHRmonize 利用大型语言模型从电子健康档案中提取医疗概念，具有提高效率、减少标注时间的潜力，但仍需要临床专业医生的监督。该框架可作为一个 Python 包提供，有望加速健康研究和改善数据整合过程。

EHRmonize: 使用大型语言模型从电子病例中抽取医学概念的框架

EHRmonize: A Framework for Medical Concept Abstraction from Electronic  Health Records using Large Language Models

Objectives: To describe the development and validation of a zero-shot
learning natural language processing (NLP) tool for abstracting data from
unstructured text contained within PDF documents, such as those found within
electronic health records. Materials and Methods: A data abstraction tool based
on the GPT-3.5 model from OpenAI was developed and compared to three physician
human abstractors in terms of time to task completion and accuracy for
abstracting data on 14 unique variables from a set of 199 de-identified radical
prostatectomy pathology reports. The reports were processed by the software
tool in vectorized and scanned formats to establish the impact of optical
character recognition on data abstraction. The tool was assessed for
superiority for data abstraction speed and non-inferiority for accuracy.
Results: The human abstractors required a mean of 101s per report for data
abstraction, with times varying from 15 to 284 s. In comparison, the software
tool required a mean of 12.8 s to process the vectorized reports and a mean of
15.8 to process the scanned reports (P < 0.001). The overall accuracies of the
three human abstractors were 94.7%, 97.8%, and 96.4% for the combined set of
2786 datapoints. The software tool had an overall accuracy of 94.2% for the
vectorized reports, proving to be non-inferior to the human abstractors at a
margin of -10% ($\alpha$=0.025). The tool had a slightly lower accuracy of
88.7% using the scanned reports, proving to be non-inferiority to 2 out of 3
human abstractors. Conclusion: The developed zero-shot learning NLP tool
affords researchers comparable levels of accuracy to that of human abstractors,
with significant time savings benefits. Because of the lack of need for
task-specific model training, the developed tool is highly generalizable and
can be used for a wide variety of data abstraction tasks, even outside the
field of medicine.

通过开智的 GPT-3.5 模型开发和比较，使用向量化和扫描格式处理 PDF 文档中的非结构化文本，从而实现了零样本学习自然语言处理工具，该工具在数据抽取速度上优于人类抽取者，并在准确性上与人类抽取者相当，具有高度通用性。