Recent developments in multimodal large language models (MLLMs) have spurred
significant interest in their potential applications across various medical
imaging domains. On the one hand, there is a temptation to use these generative
models to synthesize realistic-looking medical image data, while on the other
hand, the ability to identify synthetic image data in a pool of data is also
significantly important. In this study, we explore the potential of the Gemini
(\textit{gemini-1.0-pro-vision-latest}) and GPT-4V (gpt-4-vision-preview)
models for medical image analysis using two modalities of medical image data.
Utilizing synthetic and real imaging data, both Gemini AI and GPT-4V are first
used to classify real versus synthetic images, followed by an interpretation
and analysis of the input images. Experimental results demonstrate that both
Gemini and GPT-4 could perform some interpretation of the input images. In this
specific experiment, Gemini was able to perform slightly better than the GPT-4V
on the classification task. In contrast, responses associated with GPT-4V were
mostly generic in nature. Our early investigation presented in this work
provides insights into the potential of MLLMs to assist with the classification
and interpretation of retinal fundoscopy and lung X-ray images. We also
identify key limitations associated with the early investigation study on MLLMs
for specialized tasks in medical image analysis.

利用 Gemini 和 GPT-4V 模型，本研究尝试基于两种模态医学图像数据进行分类、解释和分析，并发现 Gemini 在分类任务上略优于 GPT-4V，而 GPT-4V 的回答主要是泛化的。该研究为多模态大型语言模型在视网膜眼底检查和肺部 X 射线图像等医学图像分析专项任务中的分类和解释提供了潜力，并识别了早期调查研究中的关键限制。

医学影像中多模态大型语言模型的实用性初探

An Early Investigation into the Utility of Multimodal Large Language  Models in Medical Imaging

A growing number of approaches exist to generate explanations for image
classification. However, few of these approaches are subjected to human-subject
evaluations, partly because it is challenging to design controlled experiments
with natural image datasets, as they leave essential factors out of the
researcher's control. With our approach, researchers can describe their desired
dataset with only a few parameters. Based on these, our library generates
synthetic image data of two 3D abstract animals. The resulting data is suitable
for algorithmic as well as human-subject evaluations. Our user study results
demonstrate that our method can create biases predictive enough for a
classifier and subtle enough to be noticeable only to every second participant
inspecting the data visually. Our approach significantly lowers the barrier for
conducting human subject evaluations, thereby facilitating more rigorous
investigations into interpretable machine learning. For our library and
datasets see, this https URL

本文提出了一种生成用于图像分类解释的合成图像数据的方法，该方法经过用户研究验证，可以降低进行人体实验的难度，并有助于研究解释可解释机器学习的更多内容。