Recently, promptable segmentation models, such as the Segment Anything Model
(SAM), have demonstrated robust zero-shot generalization capabilities on static
images. These promptable models exhibit denoising abilities for imprecise
prompt inputs, such as imprecise bounding boxes. In this paper, we explore the
potential of applying SAM to track and segment objects in videos where we
recognize the tracking task as a prompt denoising task. Specifically, we
iteratively propagate the bounding box of each object's mask in the preceding
frame as the prompt for the next frame. Furthermore, to enhance SAM's denoising
capability against position and size variations, we propose a multi-prompt
strategy where we provide multiple jittered and scaled box prompts for each
object and preserve the mask prediction with the highest semantic similarity to
the template mask. We also introduce a point-based refinement stage to handle
occlusions and reduce cumulative errors. Without involving tracking modules,
our approach demonstrates comparable performance in video object/instance
segmentation tasks on three datasets: DAVIS2017, YouTubeVOS2018, and UVO,
serving as a concise baseline and endowing SAM-based downstream applications
with tracking capabilities.

本文探讨了将 Segment Anything Model（SAM）应用于视频对象跟踪和分割任务的潜力，通过使用多种 prompt 和引入基于点的优化阶段，实现了在三个数据集上对比性能相当的视频对象 / 实例分割任务，该方法为基于 SAM 的后续应用赋予了跟踪能力。

SAM-PD: 通过提示去噪，在跟踪和分割视频中，SAM 能带我们走多远

SAM-PD: How Far Can SAM Take Us in Tracking and Segmenting Anything in  Videos by Prompt Denoising

Tracking any given object(s) spatially and temporally is a common purpose in
Visual Object Tracking (VOT) and Video Object Segmentation (VOS). Joint
tracking and segmentation have been attempted in some studies but they often
lack full compatibility of both box and mask in initialization and prediction,
and mainly focus on single-object scenarios. To address these limitations, this
paper proposes a Multi-object Mask-box Integrated framework for unified
Tracking and Segmentation, dubbed MITS. Firstly, the unified identification
module is proposed to support both box and mask reference for initialization,
where detailed object information is inferred from boxes or directly retained
from masks. Additionally, a novel pinpoint box predictor is proposed for
accurate multi-object box prediction, facilitating target-oriented
representation learning. All target objects are processed simultaneously from
encoding to propagation and decoding, as a unified pipeline for VOT and VOS.
Experimental results show MITS achieves state-of-the-art performance on both
VOT and VOS benchmarks. Notably, MITS surpasses the best prior VOT competitor
by around 6% on the GOT-10k test set, and significantly improves the
performance of box initialization on VOS benchmarks. The code is available at
this https URL

本文提出了一种多对象掩膜 - 包围盒综合框架（MITS），用于统一的跟踪和分割。该框架通过引入统一的标识模块和精确的多对象包围盒预测器，从初始化到解码实现同时处理所有目标对象的编码、传播和解码，并在 VOT 和 VOS 基准测试中取得了最先进的性能。