We present the \textbf{D}ecoupled \textbf{VI}deo \textbf{S}egmentation (DVIS)
framework, a novel approach for the challenging task of universal video
segmentation, including video instance segmentation (VIS), video semantic
segmentation (VSS), and video panoptic segmentation (VPS). Unlike previous
methods that model video segmentation in an end-to-end manner, our approach
decouples video segmentation into three cascaded sub-tasks: segmentation,
tracking, and refinement. This decoupling design allows for simpler and more
effective modeling of the spatio-temporal representations of objects,
especially in complex scenes and long videos. Accordingly, we introduce two
novel components: the referring tracker and the temporal refiner. These
components track objects frame by frame and model spatio-temporal
representations based on pre-aligned features. To improve the tracking
capability of DVIS, we propose a denoising training strategy and introduce
contrastive learning, resulting in a more robust framework named DVIS++.
Furthermore, we evaluate DVIS++ in various settings, including open vocabulary
and using a frozen pre-trained backbone. By integrating CLIP with DVIS++, we
present OV-DVIS++, the first open-vocabulary universal video segmentation
framework. We conduct extensive experiments on six mainstream benchmarks,
including the VIS, VSS, and VPS datasets. Using a unified architecture, DVIS++
significantly outperforms state-of-the-art specialized methods on these
benchmarks in both close- and open-vocabulary settings.
Code:~https://github.com/zhang-tao-whu/DVIS_Plus.

我们提出了一种新颖的 Decoupled VIdeo Segmentation (DVIS) 框架，用于解决通用视频分割的难题，包括视频实例分割 (VIS)，视频语义分割 (VSS) 和视频全景分割 (VPS)。通过将视频分割解耦成三个级联的子任务：分割，跟踪和细化，我们的方法可以更简单、更有效地建模对象的时空表示，特别是在复杂场景和长视频中。我们引入了两个新颖的组件：引用跟踪器和时序细化器，这些组件对物体进行逐帧跟踪，并基于预对齐特征建模时空表示。为了改善 DVIS 的跟踪能力，我们提出了一种去噪训练策略，并引入对比学习，从而得到更强大的名为 DVIS++ 的框架。此外，我们在多种设置下评估了 DVIS++，包括开放词汇和使用冻结预训练骨干。通过将 CLIP 与 DVIS++ 集成，我们提出了 OV-DVIS++，第一个开放词汇的通用视频分割框架。我们在包括 VIS、VSS 和 VPS 数据集在内的六个主流基准上进行了大量实验证明，在接近词汇和开放词汇的设置下，DVIS++ 在这些基准上明显优于最先进的专门方法。