Vision language models (VLMs) have recently emerged and gained the spotlight for their ability to comprehend the dual modality of image and textual data. VLMs such as LLaVA, ChatGPT-4, and Gemini have recently shown impressive performance on tasks such as natural image captioning, visual question answering (VQA), and spatial reasoning. Additionally, a universal segmentation model by Meta AI, Segment Anything Model (SAM) shows unprecedented performance at isolating objects from unforeseen images. Since medical experts, biologists, and materials scientists routinely examine microscopy or medical images in conjunction with textual information in the form of captions, literature, or reports, and draw conclusions of great importance and merit, it is indubitably essential to test the performance of VLMs and foundation models such as SAM, on these images. In this study, we charge ChatGPT, LLaVA, Gemini, and SAM with classification, segmentation, counting, and VQA tasks on a variety of microscopy images. We observe that ChatGPT and Gemini are impressively able to comprehend the visual features in microscopy images, while SAM is quite capable at isolating artefacts in a general sense. However, the performance is not close to that of a domain expert - the models are readily encumbered by the introduction of impurities, defects, artefact overlaps and diversity present in the images.

在这项研究中，我们对 ChatGPT、LLaVA、Gemini 和 SAM 进行了分类、分割、计数和 VQA 任务，发现 ChatGPT 和 Gemini 能够令人印象深刻地理解显微镜图像中的视觉特征，而 SAM 则在一般情况下具备分离伪影的能力，但性能还不如领域专家 - 这些模型在图像中存在的杂质、缺陷、伪影重叠和多样性的引入会给其带来困扰。

超越人类视角: 大型视觉语言模型在显微镜图像分析中的作用