Modern healthcare often utilises radiographic images alongside textual
reports for diagnostics, encouraging the use of Vision-Language Self-Supervised
Learning (VL-SSL) with large pre-trained models to learn versatile medical
vision representations. However, most existing VL-SSL frameworks are trained
end-to-end, which is computation-heavy and can lose vital prior information
embedded in pre-trained encoders. To address both issues, we introduce the
backbone-agnostic Adaptor framework, which preserves medical knowledge in
pre-trained image and text encoders by keeping them frozen, and employs a
lightweight Adaptor module for cross-modal learning. Experiments on medical
image classification and segmentation tasks across three datasets reveal that
our framework delivers competitive performance while cutting trainable
parameters by over 90% compared to current pre-training approaches. Notably,
when fine-tuned with just 1% of data, Adaptor outperforms several
Transformer-based methods trained on full datasets in medical image
segmentation.

现代医疗常常使用放射影像和文字报告进行诊断，鼓励使用大型预训练模型进行视觉 - 语言自监督学习 (VL-SSL) 以学习多功能的医学视觉表示。然而，大部分现有的 VL-SSL 框架都是端到端训练的，计算成本高，并且可能丢失预训练编码器中嵌入的重要先验信息。为了解决这两个问题，我们引入了适用于任意骨干网络的 Adaptor 框架，通过保持预训练的图像和文本编码器的冻结状态，并采用轻量级的 Adaptor 模块进行跨模态学习来保留医学知识。在三个数据集上进行的医学图像分类和分割任务实验表明，与当前的预训练方法相比，我们的框架在保持可训练参数减少 90% 以上的同时，具有竞争性的性能。值得注意的是，仅使用 1% 的数据进行微调时，Adaptor 在医学图像分割方面胜过了针对完整数据集训练的几种基于 Transformer 的方法。