Language-supervised pre-training has proven to be a valuable method for
extracting semantically meaningful features from images, serving as a
foundational element in multimodal systems within the computer vision and
medical imaging domains. However, resulting features are limited by the
information contained within the text. This is particularly problematic in
medical imaging, where radiologists' written findings focus on specific
observations; a challenge compounded by the scarcity of paired imaging-text
data due to concerns over leakage of personal health information. In this work,
we fundamentally challenge the prevailing reliance on language supervision for
learning general purpose biomedical imaging encoders. We introduce RAD-DINO, a
biomedical image encoder pre-trained solely on unimodal biomedical imaging data
that obtains similar or greater performance than state-of-the-art biomedical
language supervised models on a diverse range of benchmarks. Specifically, the
quality of learned representations is evaluated on standard imaging tasks
(classification and semantic segmentation), and a vision-language alignment
task (text report generation from images). To further demonstrate the drawback
of language supervision, we show that features from RAD-DINO correlate with
other medical records (e.g., sex or age) better than language-supervised
models, which are generally not mentioned in radiology reports. Finally, we
conduct a series of ablations determining the factors in RAD-DINO's
performance; notably, we observe that RAD-DINO's downstream performance scales
well with the quantity and diversity of training data, demonstrating that
image-only supervision is a scalable approach for training a foundational
biomedical image encoder.

在计算机视觉和医学图像领域的多模态系统中，语言监督的预训练被证明是从图像中提取语义有意义特征的有价值方法。然而，由于文本包含的信息有限，得到的特征受到了限制。在医学图像中，这一点尤为严重，因为放射科医生的书面结果侧重于特定观察，而相关图像 - 文本数据的稀缺性增加了挑战。本文通过在仅使用单模态医学图像数据进行预训练，引入了 RAD-DINO，一种基于生物医学图像的编码器，其性能在各种基准测试中比最先进的基于生物医学语言的监督模型取得了相似或更好的表现。