A visual-language model (VLM) pre-trained on natural images and text pairs
poses a significant barrier when applied to medical contexts due to domain
shift. Yet, adapting or fine-tuning these VLMs for medical use presents
considerable hurdles, including domain misalignment, limited access to
extensive datasets, and high-class imbalances. Hence, there is a pressing need
for strategies to effectively adapt these VLMs to the medical domain, as such
adaptations would prove immensely valuable in healthcare applications. In this
study, we propose a framework designed to adeptly tailor VLMs to the medical
domain, employing selective sampling and hard-negative mining techniques for
enhanced performance in retrieval tasks. We validate the efficacy of our
proposed approach by implementing it across two distinct VLMs: the in-domain
VLM (MedCLIP) and out-of-domain VLMs (ALBEF). We assess the performance of
these models both in their original off-the-shelf state and after undergoing
our proposed training strategies, using two extensive datasets containing
mammograms and their corresponding reports. Our evaluation spans zero-shot,
few-shot, and supervised scenarios. Through our approach, we observe a notable
enhancement in Recall@K performance for the image-text retrieval task.

本研究介绍了一种旨在将可见语言模型（VLM）巧妙地应用于医学领域的框架，采用选择性采样和难负样本挖掘技术以提升检索任务的性能。在两个不同的 VLM（MedCLIP 和 ALBEF）上实施我们的方法，并使用包含乳房 X 光和其相应报告的两个大型数据集来验证其有效性。通过我们的方法，在图像文本检索任务的 Recall@K 性能方面观察到显著的改善。

基于知识的视觉语言模型调整策略：为住院医生培训构建筛查乳腺 X 线照片的独特案例集

Knowledge-grounded Adaptation Strategy for Vision-language Models:  Building Unique Case-set for Screening Mammograms for Residents Training

The collection of data to train a Deep Learning network is costly in terms of
effort and resources. In many cases, especially in a medical context, it may
have detrimental impacts. Such as requiring invasive medical procedures or
processes which could in themselves cause medical harm. However, Deep Learning
is seen as a data hungry method. Here, we look at two commonly held adages i)
more data gives better results and ii) transfer learning will aid you when you
don't have enough data. These are widely assumed to be true and used as
evidence for choosing how to solve a problem when Deep Learning is involved. We
evaluate six medical datasets and six general datasets. Training a ResNet18
network on varying subsets of these datasets to evaluate `more data gives
better results'. We take eleven of these datasets as the sources for Transfer
Learning on subsets of the twelfth dataset -- Chest -- in order to determine
whether Transfer Learning is universally beneficial. We go further to see
whether multi-stage Transfer Learning provides a consistent benefit. Our
analysis shows that the real situation is more complex than these simple adages
-- more data could lead to a case of diminishing returns and an incorrect
choice of dataset for transfer learning can lead to worse performance, with
datasets which we would consider highly similar to the Chest dataset giving
worse results than datasets which are more dissimilar. Multi-stage transfer
learning likewise reveals complex relationships between datasets.

数据采集和深度学习网络训练消耗巨大的工作量和资源，在医疗领域尤为明显，需要侵入性的医疗程序或可能导致医学危害的过程，然而深度学习被视为数据需求量大的方法。在本研究中，我们评估了六个医疗数据集和六个通用数据集，训练 ResNet18 网络在这些数据集的不同子集上以评估 “更多数据得到更好的结果” 的观点，然后我们以十一个数据集作为转移学习的源数据，用在第十二个数据集（Chest 数据集）的不同子集上，以判断转移学习是否普遍有益。我们进一步探究多阶段转移学习是否能提供一致的益处。我们的分析表明，真实情况比这些简单的观点更复杂 —— 更多数据可能导致收益递减，选择错误的数据集进行转移学习可能会导致性能下降，而与 Chest 数据集高度相似的数据集相比，与之不太相似的数据集可能得到更差的结果。多阶段转移学习也揭示了数据集之间复杂的关系。