Segment Anything Model~(SAM), a prompt-driven foundation model for natural image segmentation, has demonstrated impressive zero-shot performance. However, SAM does not work when directly applied to medical image segmentation tasks, since SAM lacks the functionality to predict semantic labels for predicted masks and needs to provide extra prompts, such as points or boxes, to segment target regions. Meanwhile, there is a huge gap between 2D natural images and 3D medical images, so the performance of SAM is imperfect for medical image segmentation tasks. Following the above issues, we propose MaskSAM, a novel mask classification prompt-free SAM adaptation framework for medical image segmentation. We design a prompt generator combined with the image encoder in SAM to generate a set of auxiliary classifier tokens, auxiliary binary masks, and auxiliary bounding boxes. Each pair of auxiliary mask and box prompts, which can solve the requirements of extra prompts, is associated with class label predictions by the sum of the auxiliary classifier token and the learnable global classifier tokens in the mask decoder of SAM to solve the predictions of semantic labels. Meanwhile, we design a 3D depth-convolution adapter for image embeddings and a 3D depth-MLP adapter for prompt embeddings. We inject one of them into each transformer block in the image encoder and mask decoder to enable pre-trained 2D SAM models to extract 3D information and adapt to 3D medical images. Our method achieves state-of-the-art performance on AMOS2022, 90.52% Dice, which improved by 2.7% compared to nnUNet. Our method surpasses nnUNet by 1.7% on ACDC and 1.0% on Synapse datasets.

提出了MaskSAM，这是一种适用于医学图像分割的新型无需提示的SAM适应框架，通过与SAM中的图像编码器结合设计了一个提示生成器，生成一组辅助分类器令牌、辅助二进制掩模和辅助边界框，以解决额外提示的要求，同时利用3D深度卷积适配器和3D深度MLP适配器使预训练的2D SAM模型提取3D信息并适应3D医学图像。我们的方法在AMOS2022上取得了最先进的性能，Dice系数为90.52%，相比nnUNet提高了2.7%。在ACDC和Synapse数据集上，我们的方法分别超过了nnUNet 1.7%和1.0%。

MaskSAM：面向医学图像分割的自动提示SAM与掩码分类