The large-scale pre-trained vision language models (VLM) have shown remarkable domain transfer capability on natural images. However, it remains unknown whether this capability can also apply to the medical image domain. This paper thoroughly studies the knowledge transferability of pre-trained VLMs to the medical domain, where we show that well-designed medical prompts are the key to elicit knowledge from pre-trained VLMs. We demonstrate that by prompting with expressive attributes that are shared between domains, the VLM can carry the knowledge across domains and improve its generalization. This mechanism empowers VLMs to recognize novel objects with fewer or without image samples. Furthermore, to avoid the laborious manual designing process, we develop three approaches for automatic generation of medical prompts, which can inject expert-level medical knowledge and image-specific information into the prompts for fine-grained grounding. We conduct extensive experiments on thirteen different medical datasets across various modalities, showing that our well-designed prompts greatly improve the zero-shot performance compared to the default prompts, and our fine-tuned models surpass the supervised models by a significant margin.

本文通过研究预训练视觉语言模型在医学图像领域的知识传递能力，发现合理设计的医学提示语是调用预训练模型知识的关键。通过使用在领域间共享的表达属性提示，可实现知识跨越领域，改进泛化能力，对新对象的识别有所优化。此外，通过自动化生成医学提示的三种方法，可以将专家级的医学知识和图像特定信息注入提示语中，进行细粒度的信息定位，试验表明，与默认提示相比，巧妙设计的医学提示显著提高了零样本性能，且微调模型超过了受监督的模型。

使用预训练视觉语言模型的医学图像理解：一项全面研究