A visual-language model (VLM) pre-trained on natural images and text pairs
poses a significant barrier when applied to medical contexts due to domain
shift. Yet, adapting or fine-tuning these VLMs for medical use presents
considerable hurdles, including domain misalignment, limited access to
extensive datasets, and high-class imbalances. Hence, there is a pressing need
for strategies to effectively adapt these VLMs to the medical domain, as such
adaptations would prove immensely valuable in healthcare applications. In this
study, we propose a framework designed to adeptly tailor VLMs to the medical
domain, employing selective sampling and hard-negative mining techniques for
enhanced performance in retrieval tasks. We validate the efficacy of our
proposed approach by implementing it across two distinct VLMs: the in-domain
VLM (MedCLIP) and out-of-domain VLMs (ALBEF). We assess the performance of
these models both in their original off-the-shelf state and after undergoing
our proposed training strategies, using two extensive datasets containing
mammograms and their corresponding reports. Our evaluation spans zero-shot,
few-shot, and supervised scenarios. Through our approach, we observe a notable
enhancement in Recall@K performance for the image-text retrieval task.

本研究介绍了一种旨在将可见语言模型（VLM）巧妙地应用于医学领域的框架，采用选择性采样和难负样本挖掘技术以提升检索任务的性能。在两个不同的 VLM（MedCLIP 和 ALBEF）上实施我们的方法，并使用包含乳房 X 光和其相应报告的两个大型数据集来验证其有效性。通过我们的方法，在图像文本检索任务的 Recall@K 性能方面观察到显著的改善。