Segment Anything Model~(SAM), a prompt-driven foundation model for natural
image segmentation, has demonstrated impressive zero-shot performance. However,
SAM does not work when directly applied to medical image segmentation tasks,
since SAM lacks the functionality to predict semantic labels for predicted
masks and needs to provide extra prompts, such as points or boxes, to segment
target regions. Meanwhile, there is a huge gap between 2D natural images and 3D
medical images, so the performance of SAM is imperfect for medical image
segmentation tasks. Following the above issues, we propose MaskSAM, a novel
mask classification prompt-free SAM adaptation framework for medical image
segmentation. We design a prompt generator combined with the image encoder in
SAM to generate a set of auxiliary classifier tokens, auxiliary binary masks,
and auxiliary bounding boxes. Each pair of auxiliary mask and box prompts,
which can solve the requirements of extra prompts, is associated with class
label predictions by the sum of the auxiliary classifier token and the
learnable global classifier tokens in the mask decoder of SAM to solve the
predictions of semantic labels. Meanwhile, we design a 3D depth-convolution
adapter for image embeddings and a 3D depth-MLP adapter for prompt embeddings.
We inject one of them into each transformer block in the image encoder and mask
decoder to enable pre-trained 2D SAM models to extract 3D information and adapt
to 3D medical images. Our method achieves state-of-the-art performance on
AMOS2022, 90.52% Dice, which improved by 2.7% compared to nnUNet. Our method
surpasses nnUNet by 1.7% on ACDC and 1.0% on Synapse datasets.

提出了 MaskSAM，这是一种适用于医学图像分割的新型无需提示的 SAM 适应框架，通过与 SAM 中的图像编码器结合设计了一个提示生成器，生成一组辅助分类器令牌、辅助二进制掩模和辅助边界框，以解决额外提示的要求，同时利用 3D 深度卷积适配器和 3D 深度 MLP 适配器使预训练的 2D SAM 模型提取 3D 信息并适应 3D 医学图像。我们的方法在 AMOS2022 上取得了最先进的性能，Dice 系数为 90.52%，相比 nnUNet 提高了 2.7%。在 ACDC 和 Synapse 数据集上，我们的方法分别超过了 nnUNet 1.7% 和 1.0%。