The lack of large and diverse training data on Computer-Aided Diagnosis (CAD)
in breast cancer detection has been one of the concerns that impedes the
adoption of the system. Recently, pre-training with large-scale image text
datasets via Vision-Language models (VLM) (\eg CLIP) partially addresses the
issue of robustness and data efficiency in computer vision (CV). This paper
proposes Mammo-CLIP, the first VLM pre-trained on a substantial amount of
screening mammogram-report pairs, addressing the challenges of dataset
diversity and size. Our experiments on two public datasets demonstrate strong
performance in classifying and localizing various mammographic attributes
crucial for breast cancer detection, showcasing data efficiency and robustness
similar to CLIP in CV. We also propose Mammo-FActOR, a novel feature
attribution method, to provide spatial interpretation of representation with
sentence-level granularity within mammography reports. Code is available
publicly: https://github.com/batmanlab/Mammo-CLIP.

本文提出了针对乳腺癌检测中计算机辅助诊断（CAD）的大型和多样化训练数据的不足问题，通过使用大规模图像文本数据集进行预训练，通过视觉 - 语言模型（如 CLIP）部分解决了计算机视觉中鲁棒性和数据效率的问题。我们首次提出了 Mammo-CLIP，使用大量乳腺癌筛查乳房矩阵报告对其进行了预训练，解决了数据集多样性和规模的挑战。我们在两个公共数据集上的实验证明了该方法在乳腺癌检测中对关键的分类和定位属性表现出了类似 CLIP 在计算机视觉中的数据效率和鲁棒性。此外，我们还提出了 Mammo-FActOR，一种新颖的特征归因方法，可以提供在乳腺病理学报告中基于句子级粒度的空间解释。代码可在公开网址 https://github.com/batmanlab/Mammo-CLIP 上获得。

Mammo-CLIP: 一种增强乳腺摄影数据效率与鲁棒性的视觉语言基础模型

Mammo-CLIP: A Vision Language Foundation Model to Enhance Data  Efficiency and Robustness in Mammography

Although fusion of information from multiple views of mammograms plays an
important role to increase accuracy of breast cancer detection, developing
multi-view mammograms-based computer-aided diagnosis (CAD) schemes still faces
challenges and no such CAD schemes have been used in clinical practice. To
overcome the challenges, we investigate a new approach based on Contrastive
Language-Image Pre-training (CLIP), which has sparked interest across various
medical imaging tasks. By solving the challenges in (1) effectively adapting
the single-view CLIP for multi-view feature fusion and (2) efficiently
fine-tuning this parameter-dense model with limited samples and computational
resources, we introduce Mammo-CLIP, the first multi-modal framework to process
multi-view mammograms and corresponding simple texts. Mammo-CLIP uses an early
feature fusion strategy to learn multi-view relationships in four mammograms
acquired from the CC and MLO views of the left and right breasts. To enhance
learning efficiency, plug-and-play adapters are added into CLIP image and text
encoders for fine-tuning parameters and limiting updates to about 1% of the
parameters. For framework evaluation, we assembled two datasets
retrospectively. The first dataset, comprising 470 malignant and 479 benign
cases, was used for few-shot fine-tuning and internal evaluation of the
proposed Mammo-CLIP via 5-fold cross-validation. The second dataset, including
60 malignant and 294 benign cases, was used to test generalizability of
Mammo-CLIP. Study results show that Mammo-CLIP outperforms the state-of-art
cross-view transformer in AUC (0.841 vs. 0.817, 0.837 vs. 0.807) on both
datasets. It also surpasses previous two CLIP-based methods by 20.3% and 14.3%.
This study highlights the potential of applying the finetuned vision-language
models for developing next-generation, image-text-based CAD schemes of breast
cancer.

通过多视图乳房 X 光片和简单文本的多模态框架 Mammo-CLIP，本研究展示了应用微调视觉语言模型来开发下一代基于图像文本的乳腺癌 CAD 方案的潜力。