Existing vision-text contrastive learning like CLIP aims to match the paired image and caption embeddings while pushing others apart, which improves representation transferability and supports zero-shot prediction. However, medical image-text datasets are orders of magnitude below the general images and captions from the internet. Moreover, previous methods encounter many false negatives, i.e., images and reports from separate patients probably carry the same semantics but are wrongly treated as negatives. In this paper, we decouple images and texts for multimodal contrastive learning thus scaling the usable training data in a combinatorial magnitude with low cost. We also propose to replace the InfoNCE loss with semantic matching loss based on medical knowledge to eliminate false negatives in contrastive learning. We prove that MedCLIP is a simple yet effective framework: it outperforms state-of-the-art methods on zero-shot prediction, supervised classification, and image-text retrieval. Surprisingly, we observe that with only 20K pre-training data, MedCLIP wins over the state-of-the-art method (using around 200K data). Our code is available at https://github.com/RyanWangZf/MedCLIP.

本文提出了一种用于对医学图像和文本进行训练的的简单而有效的框架——MedCLIP，该框架采用了对抗学习和多模式学习，并引入医学知识语义匹配。实验结果表明，MedCLIP在零样本预测、有监督分类和图像文本检索等方面超过了现有最佳方法。令人惊讶的是，仅使用了20K的预训练数据就超过了使用约200K数据的现有最佳方法。

MedCLIP: 对未配对的医学影像和文本进行对比学习