In medical image analysis, the expertise scarcity and the high cost of data
annotation limits the development of large artificial intelligence models. This
paper investigates the potential of transfer learning with pre-trained
vision-language models (VLMs) in this domain. Currently, VLMs still struggle to
transfer to the underrepresented diseases with minimal presence and new
diseases entirely absent from the pretraining dataset. We argue that effective
adaptation of VLMs hinges on the nuanced representation learning of disease
concepts. By capitalizing on the joint visual-linguistic capabilities of VLMs,
we introduce disease-informed contextual prompting in a novel disease prototype
learning framework. This approach enables VLMs to grasp the concepts of new
disease effectively and efficiently, even with limited data. Extensive
experiments across multiple image modalities showcase notable enhancements in
performance compared to existing techniques.

通过在医学图像分析中引入疾病相关的上下文提示，利用预训练的视觉 - 语言模型（VLMs）的联合能力，提出了一种新的疾病原型学习框架，有效地提升了 VLMs 对新疾病概念的理解和性能，相较于现有方法有明显的提升。