The Segment Anything Model (SAM), a foundation model for general image
segmentation, has demonstrated impressive zero-shot performance across numerous
natural image segmentation tasks. However, SAM's performance significantly
declines when applied to medical images, primarily due to the substantial
disparity between natural and medical image domains. To effectively adapt SAM
to medical images, it is important to incorporate critical third-dimensional
information, i.e., volumetric or temporal knowledge, during fine-tuning.
Simultaneously, we aim to harness SAM's pre-trained weights within its original
2D backbone to the fullest extent. In this paper, we introduce a
modality-agnostic SAM adaptation framework, named as MA-SAM, that is applicable
to various volumetric and video medical data. Our method roots in the
parameter-efficient fine-tuning strategy to update only a small portion of
weight increments while preserving the majority of SAM's pre-trained weights.
By injecting a series of 3D adapters into the transformer blocks of the image
encoder, our method enables the pre-trained 2D backbone to extract
third-dimensional information from input data. The effectiveness of our method
has been comprehensively evaluated on four medical image segmentation tasks, by
using 10 public datasets across CT, MRI, and surgical video data. Remarkably,
without using any prompt, our method consistently outperforms various
state-of-the-art 3D approaches, surpassing nnU-Net by 0.9%, 2.6%, and 9.9% in
Dice for CT multi-organ segmentation, MRI prostate segmentation, and surgical
scene segmentation respectively. Our model also demonstrates strong
generalization, and excels in challenging tumor segmentation when prompts are
used. Our code is available at: this https URL

引入了名为 MA-SAM 的适用于各种体积和视频医学数据的模态无关 SAM 适应框架，通过在图像编码器的转换器块中注入一系列的 3D 适配器，使预训练的 2D 主干从输入数据中提取第三维信息。在四个医学图像分割任务上，使用了 CT、MRI 和外科手术视频数据的 10 个公共数据集进行全面评估，结果显示，我们的方法在不使用任何提示的情况下，始终表现优于各种最先进的三维方法，在 CT 多器官分割、MRI 前列腺分割和外科手术场景分割的 Dice 指标上分别超过 nnU-Net 0.9%、2.6% 和 9.9%。在使用提示时，我们的模型也表现出强大的泛化能力，并在具有挑战性的肿瘤分割任务中表现出色。