Panoptic segmentation has become a new standard of visual recognition task by
unifying previous semantic segmentation and instance segmentation tasks in
concert. In this paper, we propose and explore a new video extension of this
task, called video panoptic segmentation. The task requires generating
consistent panoptic segmentation as well as an association of instance ids
across video frames. To invigorate research on this new task, we present two
types of video panoptic datasets. The first is a re-organization of the
synthetic VIPER dataset into the video panoptic format to exploit its
large-scale pixel annotations. The second is a temporal extension on the
Cityscapes val. set, by providing new video panoptic annotations
(Cityscapes-VPS). Moreover, we propose a novel video panoptic segmentation
network (VPSNet) which jointly predicts object classes, bounding boxes, masks,
instance id tracking, and semantic segmentation in video frames. To provide
appropriate metrics for this task, we propose a video panoptic quality (VPQ)
metric and evaluate our method and several other baselines. Experimental
results demonstrate the effectiveness of the presented two datasets. We achieve
state-of-the-art results in image PQ on Cityscapes and also in VPQ on
Cityscapes-VPS and VIPER datasets. The datasets and code are made publicly
available.

本文提出了一种名为 Video Panoptic Segmentation 的新型视觉识别任务，同时提出了两个视频全景数据集，以及一种 VPSNet 网络，在 VPQ 度量下，在 Cityscapes-VPS 和 VIPER 数据集上实现了最先进的结果。