Driven by large data trained segmentation models, such as SAM , research in
one-shot segmentation has experienced significant advancements. Recent
contributions like PerSAM and MATCHER , presented at ICLR 2024, utilize a
similar approach by leveraging SAM with one or a few reference images to
generate high quality segmentation masks for target images. Specifically, they
utilize raw encoded features to compute cosine similarity between patches
within reference and target images along the channel dimension, effectively
generating prompt points or boxes for the target images a technique referred to
as the matching strategy. However, relying solely on raw features might
introduce biases and lack robustness for such a complex task. To address this
concern, we delve into the issues of feature interaction and uneven
distribution inherent in raw feature based matching. In this paper, we propose
a simple and training-free method to enhance the validity and robustness of the
matching strategy at no additional computational cost (NubbleDrop). The core
concept involves randomly dropping feature channels (setting them to zero)
during the matching process, thereby preventing models from being influenced by
channels containing deceptive information. This technique mimics discarding
pathological nubbles, and it can be seamlessly applied to other similarity
computing scenarios. We conduct a comprehensive set of experiments, considering
a wide range of factors, to demonstrate the effectiveness and validity of our
proposed method. Our results showcase the significant improvements achieved
through this simmple and straightforward approach.

通过在一次分割中利用大数据训练的分割模型，如 SAM，我们提出了一种简单且无需额外计算成本的方法 (NubbleDrop) 来增强匹配策略的有效性和鲁棒性，通过随机丢弃特征通道来防止模型受到具有欺骗性信息的通道的影响。

NubbleDrop：一种简单的提升提示式单次分割匹配策略的方法

NubbleDrop: A Simple Way to Improve Matching Strategy for Prompted  One-Shot Segmentation

Driven by large-data pre-training, Segment Anything Model (SAM) has been
demonstrated as a powerful and promptable framework, revolutionizing the
segmentation models. Despite the generality, customizing SAM for specific
visual concepts without man-powered prompting is under explored, e.g.,
automatically segmenting your pet dog in different images. In this paper, we
propose a training-free Personalization approach for SAM, termed as PerSAM.
Given only a single image with a reference mask, PerSAM first localizes the
target concept by a location prior, and segments it within other images or
videos via three techniques: target-guided attention, target-semantic
prompting, and cascaded post-refinement. In this way, we effectively adapt SAM
for private use without any training. To further alleviate the mask ambiguity,
we present an efficient one-shot fine-tuning variant, PerSAM-F. Freezing the
entire SAM, we introduce two learnable weights for multi-scale masks, only
training 2 parameters within 10 seconds for improved performance. To
demonstrate our efficacy, we construct a new segmentation dataset, PerSeg, for
personalized evaluation, and test our methods on video object segmentation with
competitive performance. Besides, our approach can also enhance DreamBooth to
personalize Stable Diffusion for text-to-image generation, which discards the
background disturbance for better target appearance learning. Code is released
at this https URL

本文提出了一种称为 PerSAM 的、无需训练的个性化方法，该方法首先通过位置先验定位目标概念，然后通过三种技术 - 目标引导注意力、目标语义提示和级联后处理在其他图像或视频中对其进行分割，有效地适应 SAM 的私人使用。此外，我们还提出了一种高效的单次微调变体，PerSAM-F，以缓解掩模的歧义。我们构建了一个新的分割数据集 PerSeg，并在具有竞争性的性能的视频对象分割上测试了我们的方法。