This paper presents Video K-Net, a simple, strong, and unified framework for
fully end-to-end video panoptic segmentation. The method is built upon K-Net, a
method that unifies image segmentation via a group of learnable kernels. We
observe that these learnable kernels from K-Net, which encode object
appearances and contexts, can naturally associate identical instances across
video frames. Motivated by this observation, Video K-Net learns to
simultaneously segment and track "things" and "stuff" in a video with simple
kernel-based appearance modeling and cross-temporal kernel interaction. Despite
the simplicity, it achieves state-of-the-art video panoptic segmentation
results on Citscapes-VPS, KITTI-STEP, and VIPSeg without bells and whistles. In
particular, on KITTI-STEP, the simple method can boost almost 12\% relative
improvements over previous methods. On VIPSeg, Video K-Net boosts almost 15\%
relative improvements and results in 39.8 % VPQ. We also validate its
generalization on video semantic segmentation, where we boost various baselines
by 2\% on the VSPW dataset. Moreover, we extend K-Net into clip-level video
framework for video instance segmentation, where we obtain 40.5% mAP for
ResNet50 backbone and 54.1% mAP for Swin-base on YouTube-2019 validation set.
We hope this simple, yet effective method can serve as a new, flexible baseline
in unified video segmentation design. Both code and models are released at
this https URL

本文介绍了一种名为 Video K-Net 的全新视频语义分割框架，通过与基于 kernel 的出现模型相结合来跟踪视频中的物体实例，并在多个数据集上达到了最佳表现，此外，该框架也能进一步扩展为视频示例分割和视频语义分割。