Most advances in medical image recognition supporting clinical auxiliary
diagnosis meet challenges due to the low-resource situation in the medical
field, where annotations are highly expensive and professional. This
low-resource problem can be alleviated by leveraging the transferable
representations of large-scale pre-trained vision-language models via relevant
medical text prompts. However, existing pre-trained vision-language models
require domain experts to carefully design the medical prompts, which greatly
increases the burden on clinicians. To address this problem, we propose a
weakly supervised prompt learning method MedPrompt to automatically generate
medical prompts, which includes an unsupervised pre-trained vision-language
model and a weakly supervised prompt learning model. The unsupervised
pre-trained vision-language model utilizes the natural correlation between
medical images and corresponding medical texts for pre-training, without any
manual annotations. The weakly supervised prompt learning model only utilizes
the classes of images in the dataset to guide the learning of the specific
class vector in the prompt, while the learning of other context vectors in the
prompt requires no manual annotations for guidance. To the best of our
knowledge, this is the first model to automatically generate medical prompts.
With these prompts, the pre-trained vision-language model can be freed from the
strong expert dependency of manual annotation and manual prompt design.
Experimental results show that the model using our automatically generated
prompts outperforms its full-shot learning hand-crafted prompts counterparts
with only a minimal number of labeled samples for few-shot learning, and
reaches superior or comparable accuracy on zero-shot image classification. The
proposed prompt generator is lightweight and therefore can be embedded into any
network architecture.

通过弱监督的提示学习方法 MedPrompt，自动生成医学提示并利用预训练的视觉 - 语言模型，在低资源情况下提升医学图像识别的性能和准确率。