The Depth-aware Video Panoptic Segmentation (DVPS) is a new challenging
vision problem that aims to predict panoptic segmentation and depth in a video
simultaneously. The previous work solves this task by extending the existing
panoptic segmentation method with an extra dense depth prediction and instance
tracking head. However, the relationship between the depth and panoptic
segmentation is not well explored -- simply combining existing methods leads to
competition and needs carefully weight balancing. In this paper, we present
PolyphonicFormer, a vision transformer to unify these sub-tasks under the DVPS
task and lead to more robust results. Our principal insight is that the depth
can be harmonized with the panoptic segmentation with our proposed new paradigm
of predicting instance level depth maps with object queries. Then the
relationship between the two tasks via query-based learning is explored. From
the experiments, we demonstrate the benefits of our design from both depth
estimation and panoptic segmentation aspects. Since each thing query also
encodes the instance-wise information, it is natural to perform tracking
directly with appearance learning. Our method achieves state-of-the-art results
on two DVPS datasets (Semantic KITTI, Cityscapes), and ranks 1st on the
ICCV-2021 BMTT Challenge video + depth track. Code is available at
this https URL .

本文提出了一种称为 PolyphonicFormer 的视觉变换器，它使用查询查询技术将深度估计与全景分割统一起来，具有更好的性能。通过实验证明了我们的方法在深度估计和全景分割方面取得了最先进的结果，并在 ICCV-2021 BMTT 挑战赛中获得了第一名。

PolyphonicFormer: 深度感知视频全景分割的统一查询学习

PolyphonicFormer: Unified Query Learning for Depth-aware Video Panoptic  Segmentation

In this paper, we present ViP-DeepLab, a unified model attempting to tackle
the long-standing and challenging inverse projection problem in vision, which
we model as restoring the point clouds from perspective image sequences while
providing each point with instance-level semantic interpretations. Solving this
problem requires the vision models to predict the spatial location, semantic
class, and temporally consistent instance label for each 3D point. ViP-DeepLab
approaches it by jointly performing monocular depth estimation and video
panoptic segmentation. We name this joint task as Depth-aware Video Panoptic
Segmentation, and propose a new evaluation metric along with two derived
datasets for it, which will be made available to the public. On the individual
sub-tasks, ViP-DeepLab also achieves state-of-the-art results, outperforming
previous methods by 5.1% VPQ on Cityscapes-VPS, ranking 1st on the KITTI
monocular depth estimation benchmark, and 1st on KITTI MOTS pedestrian. The
datasets and the evaluation codes are made publicly available.

ViP-DeepLab 是一种联合模型，尝试解决视觉中长期存在且具有挑战性的逆投影问题。该模型通过联合执行单眼深度估计和视频全景分割来处理该问题，并提供每个点的实例级语义解释。在子任务中，ViP-DeepLab 也取得了最先进的结果，在 Cityscapes-VPS 中超越以前的方法 5.1％ VPQ，在 KITTI 单眼深度估计基准测试中排名第一，在 KITTI MOTS 中排名第一。