Recent studies indicate that Generative Pre-trained Transformer 4 with Vision
(GPT-4V) outperforms human physicians in medical challenge tasks. However,
these evaluations primarily focused on the accuracy of multi-choice questions
alone. Our study extends the current scope by conducting a comprehensive
analysis of GPT-4V's rationales of image comprehension, recall of medical
knowledge, and step-by-step multimodal reasoning when solving New England
Journal of Medicine (NEJM) Image Challenges - an imaging quiz designed to test
the knowledge and diagnostic capabilities of medical professionals. Evaluation
results confirmed that GPT-4V outperforms human physicians regarding
multi-choice accuracy (88.0% vs. 77.0%, p=0.034). GPT-4V also performs well in
cases where physicians incorrectly answer, with over 80% accuracy. However, we
discovered that GPT-4V frequently presents flawed rationales in cases where it
makes the correct final choices (27.3%), most prominent in image comprehension
(21.6%). Regardless of GPT-4V's high accuracy in multi-choice questions, our
findings emphasize the necessity for further in-depth evaluations of its
rationales before integrating such models into clinical workflows.

通过对 Generative Pre-trained Transformer 4 with Vision (GPT-4V) 在图像理解、医学知识回忆和多模态推理等方面综合分析，本研究发现 GPT-4V 在多项选择准确性方面超过人类医生（88.0% vs. 77.0%，p=0.034），且在诊断错误的情况下准确率超过 80%。然而，我们还发现，GPT-4V 在作出正确选择的情况下，其解释经常存在缺陷（27.3%），尤其在图像理解方面（21.6%）。尽管 GPT-4V 在多项选择问题上准确率较高，但我们的发现强调了在将这类模型整合到临床工作流程之前进一步深入评估其解释的必要性。

GPT-4 视觉在医学中的专家级准确性背后的隐藏缺陷

Hidden Flaws Behind Expert-Level Accuracy of GPT-4 Vision in Medicine

Background: General-purpose large language models that utilize both text and
images have not been evaluated on a diverse array of challenging medical cases.
Methods: Using 934 cases from the NEJM Image Challenge published between 2005
and 2023, we evaluated the accuracy of the recently released Generative
Pre-trained Transformer 4 with Vision model (GPT-4V) compared to human
respondents overall and stratified by question difficulty, image type, and skin
tone. We further conducted a physician evaluation of GPT-4V on 69 NEJM
clinicopathological conferences (CPCs). Analyses were conducted for models
utilizing text alone, images alone, and both text and images.
Results: GPT-4V achieved an overall accuracy of 61% (95% CI, 58 to 64%)
compared to 49% (95% CI, 49 to 50%) for humans. GPT-4V outperformed humans at
all levels of difficulty and disagreement, skin tones, and image types; the
exception was radiographic images, where performance was equivalent between
GPT-4V and human respondents. Longer, more informative captions were associated
with improved performance for GPT-4V but similar performance for human
respondents. GPT-4V included the correct diagnosis in its differential for 80%
(95% CI, 68 to 88%) of CPCs when using text alone, compared to 58% (95% CI, 45
to 70%) of CPCs when using both images and text.
Conclusions: GPT-4V outperformed human respondents on challenging medical
cases and was able to synthesize information from both images and text, but
performance deteriorated when images were added to highly informative text.
Overall, our results suggest that multimodal AI models may be useful in medical
diagnostic reasoning but that their accuracy may depend heavily on context.

基于本研究的结果，利用大规模通用语言模型结合图像能够有效应对复杂医学病例，但图像的添加可能降低模型的准确性。