Despite the recent advances in unified image segmentation (IS), developing a
unified video segmentation (VS) model remains a challenge. This is mainly
because generic category-specified VS tasks need to detect all objects and
track them across consecutive frames, while prompt-guided VS tasks require
re-identifying the target with visual/text prompts throughout the entire video,
making it hard to handle the different tasks with the same architecture. We
make an attempt to address these issues and present a novel unified VS
architecture, namely UniVS, by using prompts as queries. UniVS averages the
prompt features of the target from previous frames as its initial query to
explicitly decode masks, and introduces a target-wise prompt cross-attention
layer in the mask decoder to integrate prompt features in the memory pool. By
taking the predicted masks of entities from previous frames as their visual
prompts, UniVS converts different VS tasks into prompt-guided target
segmentation, eliminating the heuristic inter-frame matching process. Our
framework not only unifies the different VS tasks but also naturally achieves
universal training and testing, ensuring robust performance across different
scenarios. UniVS shows a commendable balance between performance and
universality on 10 challenging VS benchmarks, covering video instance,
semantic, panoptic, object, and referring segmentation tasks. Code can be found
at https://github.com/MinghanLi/UniVS.

通过使用提示作为查询，我们提出了一种新颖的统一视频分割 (UniVS) 架构，该架构平均目标的前几帧的提示特征作为初始查询来明确解码掩码，并在掩码解码器中引入了一种目标导向的提示交叉注意力层，以将提示特征集成到内存池中。通过将先前帧中的实体的预测掩码作为其视觉提示，UniVS 将不同的视频分割任务转化为提示引导的目标分割，消除了启发式的帧间匹配过程。我们的框架不仅统一了不同的视频分割任务，还保证在不同的场景中实现了通用的训练和测试，确保了稳健的性能。UniVS 在 10 个具有挑战性的视频分割基准上展现了出色的性能和通用性，涵盖了视频实例、语义、全景、物体和引用分割任务。

UniVS：具备提示查询的统一通用视频分割

UniVS: Unified and Universal Video Segmentation with Prompts as Queries

This paper presents Video K-Net, a simple, strong, and unified framework for
fully end-to-end video panoptic segmentation. The method is built upon K-Net, a
method that unifies image segmentation via a group of learnable kernels. We
observe that these learnable kernels from K-Net, which encode object
appearances and contexts, can naturally associate identical instances across
video frames. Motivated by this observation, Video K-Net learns to
simultaneously segment and track "things" and "stuff" in a video with simple
kernel-based appearance modeling and cross-temporal kernel interaction. Despite
the simplicity, it achieves state-of-the-art video panoptic segmentation
results on Citscapes-VPS, KITTI-STEP, and VIPSeg without bells and whistles. In
particular, on KITTI-STEP, the simple method can boost almost 12\% relative
improvements over previous methods. On VIPSeg, Video K-Net boosts almost 15\%
relative improvements and results in 39.8 % VPQ. We also validate its
generalization on video semantic segmentation, where we boost various baselines
by 2\% on the VSPW dataset. Moreover, we extend K-Net into clip-level video
framework for video instance segmentation, where we obtain 40.5% mAP for
ResNet50 backbone and 54.1% mAP for Swin-base on YouTube-2019 validation set.
We hope this simple, yet effective method can serve as a new, flexible baseline
in unified video segmentation design. Both code and models are released at
this https URL

本文介绍了一种名为 Video K-Net 的全新视频语义分割框架，通过与基于 kernel 的出现模型相结合来跟踪视频中的物体实例，并在多个数据集上达到了最佳表现，此外，该框架也能进一步扩展为视频示例分割和视频语义分割。