The large-scale pre-trained vision language models (VLM) have shown
remarkable domain transfer capability on natural images. However, it remains
unknown whether this capability can also apply to the medical image domain.
This paper thoroughly studies the knowledge transferability of pre-trained VLMs
to the medical domain, where we show that well-designed medical prompts are the
key to elicit knowledge from pre-trained VLMs. We demonstrate that by prompting
with expressive attributes that are shared between domains, the VLM can carry
the knowledge across domains and improve its generalization. This mechanism
empowers VLMs to recognize novel objects with fewer or without image samples.
Furthermore, to avoid the laborious manual designing process, we develop three
approaches for automatic generation of medical prompts, which can inject
expert-level medical knowledge and image-specific information into the prompts
for fine-grained grounding. We conduct extensive experiments on thirteen
different medical datasets across various modalities, showing that our
well-designed prompts greatly improve the zero-shot performance compared to the
default prompts, and our fine-tuned models surpass the supervised models by a
significant margin.

本文通过研究预训练视觉语言模型在医学图像领域的知识传递能力，发现合理设计的医学提示语是调用预训练模型知识的关键。通过使用在领域间共享的表达属性提示，可实现知识跨越领域，改进泛化能力，对新对象的识别有所优化。此外，通过自动化生成医学提示的三种方法，可以将专家级的医学知识和图像特定信息注入提示语中，进行细粒度的信息定位，试验表明，与默认提示相比，巧妙设计的医学提示显著提高了零样本性能，且微调模型超过了受监督的模型。