We present the \textbf{D}ecoupled \textbf{VI}deo \textbf{S}egmentation (DVIS)
framework, a novel approach for the challenging task of universal video
segmentation, including video instance segmentation (VIS), video semantic
segmentation (VSS), and video panoptic segmentation (VPS). Unlike previous
methods that model video segmentation in an end-to-end manner, our approach
decouples video segmentation into three cascaded sub-tasks: segmentation,
tracking, and refinement. This decoupling design allows for simpler and more
effective modeling of the spatio-temporal representations of objects,
especially in complex scenes and long videos. Accordingly, we introduce two
novel components: the referring tracker and the temporal refiner. These
components track objects frame by frame and model spatio-temporal
representations based on pre-aligned features. To improve the tracking
capability of DVIS, we propose a denoising training strategy and introduce
contrastive learning, resulting in a more robust framework named DVIS++.
Furthermore, we evaluate DVIS++ in various settings, including open vocabulary
and using a frozen pre-trained backbone. By integrating CLIP with DVIS++, we
present OV-DVIS++, the first open-vocabulary universal video segmentation
framework. We conduct extensive experiments on six mainstream benchmarks,
including the VIS, VSS, and VPS datasets. Using a unified architecture, DVIS++
significantly outperforms state-of-the-art specialized methods on these
benchmarks in both close- and open-vocabulary settings.
Code:~https://github.com/zhang-tao-whu/DVIS_Plus.

我们提出了一种新颖的 Decoupled VIdeo Segmentation (DVIS) 框架，用于解决通用视频分割的难题，包括视频实例分割 (VIS)，视频语义分割 (VSS) 和视频全景分割 (VPS)。通过将视频分割解耦成三个级联的子任务：分割，跟踪和细化，我们的方法可以更简单、更有效地建模对象的时空表示，特别是在复杂场景和长视频中。我们引入了两个新颖的组件：引用跟踪器和时序细化器，这些组件对物体进行逐帧跟踪，并基于预对齐特征建模时空表示。为了改善 DVIS 的跟踪能力，我们提出了一种去噪训练策略，并引入对比学习，从而得到更强大的名为 DVIS++ 的框架。此外，我们在多种设置下评估了 DVIS++，包括开放词汇和使用冻结预训练骨干。通过将 CLIP 与 DVIS++ 集成，我们提出了 OV-DVIS++，第一个开放词汇的通用视频分割框架。我们在包括 VIS、VSS 和 VPS 数据集在内的六个主流基准上进行了大量实验证明，在接近词汇和开放词汇的设置下，DVIS++ 在这些基准上明显优于最先进的专门方法。

DVIS++: 通用视频分割的改进解耦框架

DVIS++: Improved Decoupled Framework for Universal Video Segmentation

Training data for video segmentation are expensive to annotate. This impedes
extensions of end-to-end algorithms to new video segmentation tasks, especially
in large-vocabulary settings. To 'track anything' without training on video
data for every individual task, we develop a decoupled video segmentation
approach (DEVA), composed of task-specific image-level segmentation and
class/task-agnostic bi-directional temporal propagation. Due to this design, we
only need an image-level model for the target task (which is cheaper to train)
and a universal temporal propagation model which is trained once and
generalizes across tasks. To effectively combine these two modules, we use
bi-directional propagation for (semi-)online fusion of segmentation hypotheses
from different frames to generate a coherent segmentation. We show that this
decoupled formulation compares favorably to end-to-end approaches in several
data-scarce tasks including large-vocabulary video panoptic segmentation,
open-world video segmentation, referring video segmentation, and unsupervised
video object segmentation. Code is available at:
this https URL

开发了一种解耦视频分割方法（DEVA），包括任务特定的图像级分割和与类别 / 任务无关的双向时间传播模型，通过在线融合来生成一致的分割，在大词汇视频分割、开放世界视频分割、参考视频分割和无监督视频目标分割等多个数据稀缺任务中与端到端方法相比具有优势。