Zero-shot medical image classification is a critical process in real-world
scenarios where we have limited access to all possible diseases or large-scale
annotated data. It involves computing similarity scores between a query medical
image and possible disease categories to determine the diagnostic result.
Recent advances in pretrained vision-language models (VLMs) such as CLIP have
shown great performance for zero-shot natural image recognition and exhibit
benefits in medical applications. However, an explainable zero-shot medical
image recognition framework with promising performance is yet under
development. In this paper, we propose a novel CLIP-based zero-shot medical
image classification framework supplemented with ChatGPT for explainable
diagnosis, mimicking the diagnostic process performed by human experts. The key
idea is to query large language models (LLMs) with category names to
automatically generate additional cues and knowledge, such as disease symptoms
or descriptions other than a single category name, to help provide more
accurate and explainable diagnosis in CLIP. We further design specific prompts
to enhance the quality of generated texts by ChatGPT that describe visual
medical features. Extensive results on one private dataset and four public
datasets along with detailed analysis demonstrate the effectiveness and
explainability of our training-free zero-shot diagnosis pipeline, corroborating
the great potential of VLMs and LLMs for medical applications.

该研究提出了一种基于 CLIP 和 ChatGPT 的零样本医学图像分类框架，使用大型语言模型自动生成额外的线索和知识以进行更准确和可解释的诊断，展示了 VLM 和 LLM 在医疗应用中的巨大潜力。