Driven by the large foundation models, the development of artificial
intelligence has witnessed tremendous progress lately, leading to a surge of
general interest from the public. In this study, we aim to assess the
performance of OpenAI's newest model, GPT-4V(ision), specifically in the realm
of multimodal medical diagnosis. Our evaluation encompasses 17 human body
systems, including Central Nervous System, Head and Neck, Cardiac, Chest,
Hematology, Hepatobiliary, Gastrointestinal, Urogenital, Gynecology,
Obstetrics, Breast, Musculoskeletal, Spine, Vascular, Oncology, Trauma,
Pediatrics, with images taken from 8 modalities used in daily clinic routine,
e.g., X-ray, Computed Tomography (CT), Magnetic Resonance Imaging (MRI),
Positron Emission Tomography (PET), Digital Subtraction Angiography (DSA),
Mammography, Ultrasound, and Pathology. We probe the GPT-4V's ability on
multiple clinical tasks with or without patent history provided, including
imaging modality and anatomy recognition, disease diagnosis, report generation,
disease localisation.
Our observation shows that, while GPT-4V demonstrates proficiency in
distinguishing between medical image modalities and anatomy, it faces
significant challenges in disease diagnosis and generating comprehensive
reports. These findings underscore that while large multimodal models have made
significant advancements in computer vision and natural language processing, it
remains far from being used to effectively support real-world medical
applications and clinical decision-making.
All images used in this report can be found in
this https URL

通过评估 OpenAI 最新模型 GPT-4V (ision) 在多模态医学诊断领域中的表现，我们发现尽管 GPT-4V 在区分医学图像模态和解剖学方面表现出色，但在疾病诊断和生成综合报告方面面临重大挑战，这表明尽管大型多模态模型在计算机视觉和自然语言处理方面取得了重大进展，但其在有效支持现实医疗应用和临床决策方面仍有很大差距。