In this paper, we introduce semi-supervised video object segmentation (VOS)
to panoptic wild scenes and present a large-scale benchmark as well as a
baseline method for it. Previous benchmarks for VOS with sparse annotations are
not sufficient to train or evaluate a model that needs to process all possible
objects in real-world scenarios. Our new benchmark (VIPOSeg) contains
exhaustive object annotations and covers various real-world object categories
which are carefully divided into subsets of thing/stuff and seen/unseen classes
for comprehensive evaluation. Considering the challenges in panoptic VOS, we
propose a strong baseline method named panoptic object association with
transformers (PAOT), which uses panoptic identification to associate objects
with a pyramid architecture on multiple scales. Experimental results show that
VIPOSeg can not only boost the performance of VOS models by panoptic training
but also evaluate them comprehensively in panoptic scenes. Previous methods for
classic VOS still need to improve in performance and efficiency when dealing
with panoptic scenes, while our PAOT achieves SOTA performance with good
efficiency on VIPOSeg and previous VOS benchmarks. PAOT also ranks 1st in the
VOT2022 challenge. Our dataset is available at
this https URL

本文提出了半监督视频对象分割的概念，给出了 Panoptic Wild Scenes 的大型基准和基线方法，建立了 VIPOSeg 数据集，提出了基于当下流行的 Transformer 的网格架构的强基线方法 PAOT，在 VIPOSeg 和其他 VOS 基准中取得了 SOTA 的性能，并在 VOT2022 挑战活动中获得第一名。