Contrastive Language-Image Pre-training (CLIP), a straightforward yet
effective pre-training paradigm, successfully introduces semantic-rich text
supervision to vision models and has demonstrated promising results in various
tasks due to its generalizability and interpretability. It has recently gained
increasing interest in the medical imaging domain, either as a powerful
pre-training paradigm for medical vision language alignment or a pre-trained
key component for various clinical tasks. With the aim of facilitating a deeper
understanding of this promising direction, this survey offers an in-depth
exploration of the CLIP paradigm within the domain of medical imaging,
regarding both refined CLIP pre-training and CLIP-driven applications. Our
survey (1) starts with a brief introduction to the fundamentals of CLIP
methodology. (2) Then, we investigate the adaptation of CLIP pre-training in
the medical domain, focusing on how to optimize CLIP given characteristics of
medical images and reports. (3) Furthermore, we explore the practical
utilization of CLIP pre-trained models in various tasks, including
classification, dense prediction, and cross-modal tasks. (4) Finally, we
discuss existing limitations of CLIP in the context of medical imaging and
propose forward-looking directions to address the demands of medical imaging
domain. We expect that this comprehensive survey will provide researchers in
the field of medical image analysis with a holistic understanding of the CLIP
paradigm and its potential implications. The project page is available at
this https URL, which will be
regularly updated.

通过对 Contrastive Language-Image Pre-training (CLIP) 在医学成像领域中的深入探索，本综述论文旨在为医学图像分析领域的研究人员提供对 CLIP 范式及其潜在影响的整体理解。

医学影像中的 CLIP 技术综述

CLIP in Medical Imaging: A Comprehensive Survey

Recent work has shown that self-supervised pre-training leads to improvements
over supervised learning on challenging visual recognition tasks. CLIP, an
exciting new approach to learning with language supervision, demonstrates
promising performance on a wide variety of benchmarks. In this work, we explore
whether self-supervised learning can aid in the use of language supervision for
visual representation learning. We introduce SLIP, a multi-task learning
framework for combining self-supervised learning and CLIP pre-training. After
pre-training with Vision Transformers, we thoroughly evaluate representation
quality and compare performance to both CLIP and self-supervised learning under
three distinct settings: zero-shot transfer, linear classification, and
end-to-end finetuning. Across ImageNet and a battery of additional datasets, we
find that SLIP improves accuracy by a large margin. We validate our results
further with experiments on different model sizes, training schedules, and
pre-training datasets. Our findings show that SLIP enjoys the best of both
worlds: better performance than self-supervision (+8.1% linear accuracy) and
language supervision (+5.2% zero-shot accuracy).

本文提出 SLIP，结合基于自监督的学习和 CLIP 预训练的多任务学习框架，通过在 ImageNet 和其他数据集上进行多个实验，发现 SLIP 表现出更好的性能，同时获得比基于自监督学习和语言监督学习更高的准确性提高。