Large language models (LLMs) have demonstrated remarkable capabilities in
natural language understanding and generation across various domains, including
medicine. We present a comprehensive evaluation of GPT-4, a state-of-the-art
LLM, on medical competency examinations and benchmark datasets. GPT-4 is a
general-purpose model that is not specialized for medical problems through
training or engineered to solve clinical tasks. Our analysis covers two sets of
official practice materials for the USMLE, a three-step examination program
used to assess clinical competency and grant licensure in the United States. We
also evaluate performance on the MultiMedQA suite of benchmark datasets. Beyond
measuring model performance, experiments were conducted to investigate the
influence of test questions containing both text and images on model
performance, probe for memorization of content during training, and study
probability calibration, which is of critical importance in high-stakes
applications like medicine. Our results show that GPT-4, without any
specialized prompt crafting, exceeds the passing score on USMLE by over 20
points and outperforms earlier general-purpose models (GPT-3.5) as well as
models specifically fine-tuned on medical knowledge (Med-PaLM, a prompt-tuned
version of Flan-PaLM 540B). In addition, GPT-4 is significantly better
calibrated than GPT-3.5, demonstrating a much-improved ability to predict the
likelihood that its answers are correct. We also explore the behavior of the
model qualitatively through a case study that shows the ability of GPT-4 to
explain medical reasoning, personalize explanations to students, and
interactively craft new counterfactual scenarios around a medical case.
Implications of the findings are discussed for potential uses of GPT-4 in
medical education, assessment, and clinical practice, with appropriate
attention to challenges of accuracy and safety.

通过对 USMLE 和 MultiMedQA 基准数据集的全面评估，我们发现不需要专门的提示造型来激发 GPT-4，它的表现超过了 USMLE 的合格分数约 20 分，并表现优于早期的通用模型（GPT-3.5）以及专门针对医学知识进行细化调整的模型（Med-PaLM，Flan-PaLM540B 的提示调整版本）。