In this study, we focus on building up a model that can Segment Anything in
medical scenarios, driven by Text prompts, termed as SAT. Our main
contributions are three folds: (i) on data construction, we combine multiple
knowledge sources to construct a multi-modal medical knowledge tree; Then we
build up a large-scale segmentation dataset for training, by collecting over
11K 3D medical image scans from 31 segmentation datasets with careful
standardization on both visual scans and label space; (ii) on model training,
we formulate a universal segmentation model, that can be prompted by inputting
medical terminologies in text form. We present a knowledge-enhanced
representation learning framework, and a series of strategies for effectively
training on the combination of a large number of datasets; (iii) on model
evaluation, we train a SAT-Nano with only 107M parameters, to segment 31
different segmentation datasets with text prompt, resulting in 362 categories.
We thoroughly evaluate the model from three aspects: averaged by body regions,
averaged by classes, and average by datasets, demonstrating comparable
performance to 36 specialist nnUNets, i.e., we train nnUNet models on each
dataset/subset, resulting in 36 nnUNets with around 1000M parameters for the 31
datasets. We will release all the codes, and models used in this report, i.e.,
SAT-Nano. Moreover, we will offer SAT-Ultra in the near future, which is
trained with model of larger size, on more diverse datasets. Webpage URL:
this https URL

本研究中，我们建立了一个可以通过文本提示来在医疗场景中对任意物体进行分割的模型，称之为 SAT。我们的主要贡献有三个方面：（一）在数据构造方面，我们整合了多个知识源来构建一个多模态的医学知识树，并通过收集超过 11,000 个来自 31 个分割数据集的三维医学图像扫描的大规模分割数据集进行训练，同时对图像扫描和标签空间进行了精确标准化；（二）在模型训练方面，我们提出了一种通用的分割模型，可以通过输入文本形式的医学术语进行提示。我们提出了一种知识增强的表示学习框架以及一系列有效训练策略，结合了大量数据集进行训练；（三）在模型评估方面，我们用只有 107M 参数的 SAT-Nano 模型对 31 个不同的分割数据集进行了分割，使用文本提示，得到了 362 个类别。我们从身体区域的平均值、类别的平均值和数据集的平均值三个方面对模型进行了全面评估，结果表明与 36 个专家 nnUNets 的性能相当，即我们对每个数据集 / 子集训练 nnUNet 模型，为 31 个数据集训练了 36 个大约有 1,000M 参数的 nnUNets 模型。我们将在本报告中公布所有用到的代码和模型，即 SAT-Nano，并且我们将在不久的将来提供 SAT-Ultra，它将在更多的多样化数据集上使用更大尺寸的模型进行训练。

一种通用的基于文本提示的医学图像分割模型

One Model to Rule them All: Towards Universal Segmentation for Medical  Images with Text Prompts

Segment Anything Model (SAM) has achieved impressive results for natural
image segmentation with input prompts such as points and bounding boxes. Its
success largely owes to massive labeled training data. However, directly
applying SAM to medical image segmentation cannot perform well because SAM
lacks medical knowledge -- it does not use medical images for training. To
incorporate medical knowledge into SAM, we introduce SA-Med2D-20M, a
large-scale segmentation dataset of 2D medical images built upon numerous
public and private datasets. It consists of 4.6 million 2D medical images and
19.7 million corresponding masks, covering almost the whole body and showing
significant diversity. This paper describes all the datasets collected in
SA-Med2D-20M and details how to process these datasets. Furthermore,
comprehensive statistics of SA-Med2D-20M are presented to facilitate the better
use of our dataset, which can help the researchers build medical vision
foundation models or apply their models to downstream medical applications. We
hope that the large scale and diversity of SA-Med2D-20M can be leveraged to
develop medical artificial intelligence for enhancing diagnosis, medical image
analysis, knowledge sharing, and education. The data with the redistribution
license is publicly available at this https URL

通过在 Segment Anything Model（SAM）中引入医学知识，我们介绍了 SA-Med2D-20M，这是一个基于众多公共和私有数据集构建的大规模分割数据集，包含了 460 万个 2D 医学图像和 1970 万个相应的蒙版，并详细描述了该数据集的构建和处理方法。我们希望利用 SA-Med2D-20M 数据集的大规模和多样性来开发医学人工智能，以增强诊断、医学图像分析、知识共享和教育。