Depth-aware panoptic segmentation is an emerging topic in computer vision
which combines semantic and geometric understanding for more robust scene
interpretation. Recent works pursue unified frameworks to tackle this challenge
but mostly still treat it as two individual learning tasks, which limits their
potential for exploring cross-domain information. We propose a deeply unified
framework for depth-aware panoptic segmentation, which performs joint
segmentation and depth estimation both in a per-segment manner with identical
object queries. To narrow the gap between the two tasks, we further design a
geometric query enhancement method, which is able to integrate scene geometry
into object queries using latent representations. In addition, we propose a
bi-directional guidance learning approach to facilitate cross-task feature
learning by taking advantage of their mutual relations. Our method sets the new
state of the art for depth-aware panoptic segmentation on both Cityscapes-DVPS
and SemKITTI-DVPS datasets. Moreover, our guidance learning approach is shown
to deliver performance improvement even under incomplete supervision labels.

深度感知全景分割是计算机视觉中的新兴话题，结合了语义和几何理解，以实现更稳健的场景解释。我们提出了一种深度统一框架，用于深度感知全景分割，通过相同的对象查询以分段方式执行联合分割和深度估计。为了缩小两个任务之间的差距，我们进一步设计了一种几何查询增强方法，利用潜在表示将场景几何集成到对象查询中。此外，我们提出了一种双向引导学习方法，通过利用它们的相互关系来促进跨任务特征学习。我们的方法在 Cityscapes-DVPS 和 SemKITTI-DVPS 数据集上树立了深度感知全景分割的最新技术水平，并且我们的引导学习方法还在不完整监督标签下提高了性能。

深度统一的面向深度感知全景分割与双向引导学习

Towards Deeply Unified Depth-aware Panoptic Segmentation with  Bi-directional Guidance Learning

This paper presents a unified framework for depth-aware panoptic segmentation
(DPS), which aims to reconstruct 3D scene with instance-level semantics from
one single image. Prior works address this problem by simply adding a dense
depth regression head to panoptic segmentation (PS) networks, resulting in two
independent task branches. This neglects the mutually-beneficial relations
between these two tasks, thus failing to exploit handy instance-level semantic
cues to boost depth accuracy while also producing sub-optimal depth maps. To
overcome these limitations, we propose a unified framework for the DPS task by
applying a dynamic convolution technique to both the PS and depth prediction
tasks. Specifically, instead of predicting depth for all pixels at a time, we
generate instance-specific kernels to predict depth and segmentation masks for
each instance. Moreover, leveraging the instance-wise depth estimation scheme,
we add additional instance-level depth cues to assist with supervising the
depth learning via a new depth loss. Extensive experiments on Cityscapes-DPS
and SemKITTI-DPS show the effectiveness and promise of our method. We hope our
unified solution to DPS can lead a new paradigm in this area. Code is available
at this https URL

本文提出了一种深度感知全景分割 (DPS) 的统一框架，该框架通过应用动态卷积技术来解决全景分割和深度预测任务之间的互补关系，从而实现对单张图像进行实例级语义重建，通过实例级深度估计方案，为辅助监督深度学习而添加额外的实例级深度提示，实验结果表明该方法具有实用性和前景。