The scarcity of annotated data has sparked significant interest in unsupervised pre-training methods that leverage medical reports as auxiliary signals for medical visual representation learning. However, existing research overlooks the multi-granularity nature of medical visual representation and lacks suitable contrastive learning techniques to improve the models' generalizability across different granularities, leading to the underutilization of image-text information. To address this, we propose MLIP, a novel framework leveraging domain-specific medical knowledge as guiding signals to integrate language information into the visual domain through image-text contrastive learning. Our model includes global contrastive learning with our designed divergence encoder, local token-knowledge-patch alignment contrastive learning, and knowledge-guided category-level contrastive learning with expert knowledge. Experimental evaluations reveal the efficacy of our model in enhancing transfer performance for tasks such as image classification, object detection, and semantic segmentation. Notably, MLIP surpasses state-of-the-art methods even with limited annotated data, highlighting the potential of multimodal pre-training in advancing medical representation learning.

提出了一种利用领域特定的医学知识作为引导信号，在医学视觉表示中将语言信息整合到视觉领域的MLIP框架，通过图像-文本对比学习来改善模型在不同粒度上的泛化性能。实验证实，即使在有限的标注数据情况下，MLIP也胜过最先进的方法，凸显了多模态预训练在推进医学表示学习中的潜力。

MLIP：利用分歧编码器和知识引导的对比学习增强医学视觉表达