Despite the recent advances in unified image segmentation (IS), developing a
unified video segmentation (VS) model remains a challenge. This is mainly
because generic category-specified VS tasks need to detect all objects and
track them across consecutive frames, while prompt-guided VS tasks require
re-identifying the target with visual/text prompts throughout the entire video,
making it hard to handle the different tasks with the same architecture. We
make an attempt to address these issues and present a novel unified VS
architecture, namely UniVS, by using prompts as queries. UniVS averages the
prompt features of the target from previous frames as its initial query to
explicitly decode masks, and introduces a target-wise prompt cross-attention
layer in the mask decoder to integrate prompt features in the memory pool. By
taking the predicted masks of entities from previous frames as their visual
prompts, UniVS converts different VS tasks into prompt-guided target
segmentation, eliminating the heuristic inter-frame matching process. Our
framework not only unifies the different VS tasks but also naturally achieves
universal training and testing, ensuring robust performance across different
scenarios. UniVS shows a commendable balance between performance and
universality on 10 challenging VS benchmarks, covering video instance,
semantic, panoptic, object, and referring segmentation tasks. Code can be found
at https://github.com/MinghanLi/UniVS.

通过使用提示作为查询，我们提出了一种新颖的统一视频分割 (UniVS) 架构，该架构平均目标的前几帧的提示特征作为初始查询来明确解码掩码，并在掩码解码器中引入了一种目标导向的提示交叉注意力层，以将提示特征集成到内存池中。通过将先前帧中的实体的预测掩码作为其视觉提示，UniVS 将不同的视频分割任务转化为提示引导的目标分割，消除了启发式的帧间匹配过程。我们的框架不仅统一了不同的视频分割任务，还保证在不同的场景中实现了通用的训练和测试，确保了稳健的性能。UniVS 在 10 个具有挑战性的视频分割基准上展现了出色的性能和通用性，涵盖了视频实例、语义、全景、物体和引用分割任务。

UniVS：具备提示查询的统一通用视频分割

UniVS: Unified and Universal Video Segmentation with Prompts as Queries

Continual pre-training has been urgent for adapting a pre-trained model to a
multitude of domains and tasks in the fast-evolving world. In practice, a
continually pre-trained model is expected to demonstrate not only greater
capacity when fine-tuned on pre-trained domains but also a non-decreasing
performance on unseen ones. In this work, we first investigate such anytime
fine-tuning effectiveness of existing continual pre-training approaches,
concluding with unanimously decreased performance on unseen domains. To this
end, we propose a prompt-guided continual pre-training method, where we train a
hypernetwork to generate domain-specific prompts by both agreement and
disagreement losses. The agreement loss maximally preserves the generalization
of a pre-trained model to new domains, and the disagreement one guards the
exclusiveness of the generated hidden states for each domain. Remarkably,
prompts by the hypernetwork alleviate the domain identity when fine-tuning and
promote knowledge transfer across domains. Our method achieved improvements of
3.57% and 3.4% on two real-world datasets (including domain shift and temporal
shift), respectively, demonstrating its efficacy.

通过超网络生成领域特定的提示，以解决一直持续预训练方法在新领域性能下降的问题，并促进跨领域的知识迁移。该方法在两个真实数据集上实现了 3.57% 和 3.4% 的改进，证明其有效性。