Recent studies have demonstrated promising performance of ChatGPT and GPT-4
on several medical domain tasks. However, none have assessed its performance
using a large-scale real-world electronic health record database, nor have
evaluated its utility in providing clinical diagnostic assistance for patients
across a full range of disease presentation. We performed two analyses using
ChatGPT and GPT-4, one to identify patients with specific medical diagnoses
using a real-world large electronic health record database and the other, in
providing diagnostic assistance to healthcare workers in the prospective
evaluation of hypothetical patients. Our results show that GPT-4 across disease
classification tasks with chain of thought and few-shot prompting can achieve
performance as high as 96% F1 scores. For patient assessment, GPT-4 can
accurately diagnose three out of four times. However, there were mentions of
factually incorrect statements, overlooking crucial medical findings,
recommendations for unnecessary investigations and overtreatment. These issues
coupled with privacy concerns, make these models currently inadequate for real
world clinical use. However, limited data and time needed for prompt
engineering in comparison to configuration of conventional machine learning
workflows highlight their potential for scalability across healthcare
applications.

在使用真实的大型电子病历数据库进行两项分析后，发现 ChatGPT 和 GPT-4 可以通过思路链和几次提示，准确地完成疾病分类任务，并为卫生保健工作者提供诊断辅助，但是这些模型目前存在错误陈述、忽视重要医学发现、推荐不必要的调查和过度治疗等问题，并伴随有隐私问题，因此仍不适用于现实世界的临床使用。不过，与传统机器学习工作流程的配置相比，这些模型所需的数据和时间较少，突出了它们在卫生保健应用中的可扩展性潜力。