Segment anything models (SAMs) are gaining attention for their zero-shot
generalization capability in segmenting objects of unseen classes and in unseen
domains when properly prompted. Interactivity is a key strength of SAMs,
allowing users to iteratively provide prompts that specify objects of interest
to refine outputs. However, to realize the interactive use of SAMs for 3D
medical imaging tasks, rapid inference times are necessary. High memory
requirements and long processing delays remain constraints that hinder the
adoption of SAMs for this purpose. Specifically, while 2D SAMs applied to 3D
volumes contend with repetitive computation to process all slices
independently, 3D SAMs suffer from an exponential increase in model parameters
and FLOPS. To address these challenges, we present FastSAM3D which accelerates
SAM inference to 8 milliseconds per 128*128*128 3D volumetric image on an
NVIDIA A100 GPU. This speedup is accomplished through 1) a novel layer-wise
progressive distillation scheme that enables knowledge transfer from a complex
12-layer ViT-B to a lightweight 6-layer ViT-Tiny variant encoder without
training from scratch; and 2) a novel 3D sparse flash attention to replace
vanilla attention operators, substantially reducing memory needs and improving
parallelization. Experiments on three diverse datasets reveal that FastSAM3D
achieves a remarkable speedup of 527.38x compared to 2D SAMs and 8.75x compared
to 3D SAMs on the same volumes without significant performance decline. Thus,
FastSAM3D opens the door for low-cost truly interactive SAM-based 3D medical
imaging segmentation with commonly used GPU hardware. Code is available at
this https URL

快速交互式的基于 SAM 的三维医学图像分割方法 FastSAM3D 通过层次递进提取复杂 12 层 ViT-B 和轻量级 6 层 ViT-Tiny 变种编码器之间的知识传递，并使用稀疏闪光注意力代替传统的注意力操作符，大幅减少内存需求和提高并行化，从而在相同体积上与 2D SAMs 相比，实现了 527.38 倍的加速，在与 3D SAMs 相比，实现了 8.75 倍的加速，并且性能下降不明显，为常用 GPU 硬件实现低成本、真正交互式的基于 SAM 的三维医学图像分割开辟了道路。