In this paper, we present ViP-DeepLab, a unified model attempting to tackle
the long-standing and challenging inverse projection problem in vision, which
we model as restoring the point clouds from perspective image sequences while
providing each point with instance-level semantic interpretations. Solving this
problem requires the vision models to predict the spatial location, semantic
class, and temporally consistent instance label for each 3D point. ViP-DeepLab
approaches it by jointly performing monocular depth estimation and video
panoptic segmentation. We name this joint task as Depth-aware Video Panoptic
Segmentation, and propose a new evaluation metric along with two derived
datasets for it, which will be made available to the public. On the individual
sub-tasks, ViP-DeepLab also achieves state-of-the-art results, outperforming
previous methods by 5.1% VPQ on Cityscapes-VPS, ranking 1st on the KITTI
monocular depth estimation benchmark, and 1st on KITTI MOTS pedestrian. The
datasets and the evaluation codes are made publicly available.

ViP-DeepLab 是一种联合模型，尝试解决视觉中长期存在且具有挑战性的逆投影问题。该模型通过联合执行单眼深度估计和视频全景分割来处理该问题，并提供每个点的实例级语义解释。在子任务中，ViP-DeepLab 也取得了最先进的结果，在 Cityscapes-VPS 中超越以前的方法 5.1％ VPQ，在 KITTI 单眼深度估计基准测试中排名第一，在 KITTI MOTS 中排名第一。