Background: General-purpose large language models that utilize both text and
images have not been evaluated on a diverse array of challenging medical cases.
Methods: Using 934 cases from the NEJM Image Challenge published between 2005
and 2023, we evaluated the accuracy of the recently released Generative
Pre-trained Transformer 4 with Vision model (GPT-4V) compared to human
respondents overall and stratified by question difficulty, image type, and skin
tone. We further conducted a physician evaluation of GPT-4V on 69 NEJM
clinicopathological conferences (CPCs). Analyses were conducted for models
utilizing text alone, images alone, and both text and images.
Results: GPT-4V achieved an overall accuracy of 61% (95% CI, 58 to 64%)
compared to 49% (95% CI, 49 to 50%) for humans. GPT-4V outperformed humans at
all levels of difficulty and disagreement, skin tones, and image types; the
exception was radiographic images, where performance was equivalent between
GPT-4V and human respondents. Longer, more informative captions were associated
with improved performance for GPT-4V but similar performance for human
respondents. GPT-4V included the correct diagnosis in its differential for 80%
(95% CI, 68 to 88%) of CPCs when using text alone, compared to 58% (95% CI, 45
to 70%) of CPCs when using both images and text.
Conclusions: GPT-4V outperformed human respondents on challenging medical
cases and was able to synthesize information from both images and text, but
performance deteriorated when images were added to highly informative text.
Overall, our results suggest that multimodal AI models may be useful in medical
diagnostic reasoning but that their accuracy may depend heavily on context.

基于本研究的结果，利用大规模通用语言模型结合图像能够有效应对复杂医学病例，但图像的添加可能降低模型的准确性。