The Segment Anything Model (SAM) marks a notable milestone in segmentation
models, highlighted by its robust zero-shot capabilities and ability to handle
diverse prompts. SAM follows a pipeline that separates interactive segmentation
into image preprocessing through a large encoder and interactive inference via
a lightweight decoder, ensuring efficient real-time performance. However, SAM
faces stability issues in challenging samples upon this pipeline. These issues
arise from two main factors. Firstly, the image preprocessing disables SAM from
dynamically using image-level zoom-in strategies to refocus on the target
object during interaction. Secondly, the lightweight decoder struggles to
sufficiently integrate interactive information with image embeddings. To
address these two limitations, we propose FocSAM with a pipeline redesigned on
two pivotal aspects. First, we propose Dynamic Window Multi-head Self-Attention
(Dwin-MSA) to dynamically refocus SAM's image embeddings on the target object.
Dwin-MSA localizes attention computations around the target object, enhancing
object-related embeddings with minimal computational overhead. Second, we
propose Pixel-wise Dynamic ReLU (P-DyReLU) to enable sufficient integration of
interactive information from a few initial clicks that have significant impacts
on the overall segmentation results. Experimentally, FocSAM augments SAM's
interactive segmentation performance to match the existing state-of-the-art
method in segmentation quality, requiring only about 5.6% of this method's
inference time on CPUs.

FocSAM 通过 Dynamic Window Multi-head Self-Attention 和 Pixel-wise Dynamic ReLU 的优化来提高 Segment Anything Model (SAM) 在交互式分割方面的性能，使其与现有的分割质量最先进的方法相匹配，但只需使用该方法推理时间的约 5.6%。