An increasing number of public datasets have shown a marked impact on
automated organ segmentation and tumor detection. However, due to the small
size and partially labeled problem of each dataset, as well as a limited
investigation of diverse types of tumors, the resulting models are often
limited to segmenting specific organs/tumors and ignore the semantics of
anatomical structures, nor can they be extended to novel domains. To address
these issues, we propose the CLIP-Driven Universal Model, which incorporates
text embedding learned from Contrastive Language-Image Pre-training (CLIP) to
segmentation models. This CLIP-based label encoding captures anatomical
relationships, enabling the model to learn a structured feature embedding and
segment 25 organs and 6 types of tumors. The proposed model is developed from
an assembly of 14 datasets, using a total of 3,410 CT scans for training and
then evaluated on 6,162 external CT scans from 3 additional datasets. We rank
first on the Medical Segmentation Decathlon (MSD) public leaderboard and
achieve state-of-the-art results on Beyond The Cranial Vault (BTCV).
Additionally, the Universal Model is computationally more efficient (6x faster)
compared with dataset-specific models, generalized better to CT scans from
varying sites, and shows stronger transfer learning performance on novel tasks.

本文介绍了 CLIP-Driven Universal Model 模型，它基于 Contrastive Language-Image Pre-training 学习的文本嵌入，结合分割模型，可以分割 25 个器官和 6 种肿瘤，具有更高的泛化性能和更高的计算效率。