We present an end-to-end joint training framework that explicitly models
6-DoF motion of multiple dynamic objects, ego-motion and depth in a monocular
camera setup without supervision. Our technical contributions are three-fold.
First, we highlight the fundamental difference between inverse and forward
projection while modeling the individual motion of each rigid object, and
propose a geometrically correct projection pipeline using a neural forward
projection module. Second, we design a unified instance-aware photometric and
geometric consistency loss that holistically imposes self-supervisory signals
for every background and object region. Lastly, we introduce a general-purpose
auto-annotation scheme using any off-the-shelf instance segmentation and
optical flow models to produce video instance segmentation maps that will be
utilized as input to our training pipeline. These proposed elements are
validated in a detailed ablation study. Through extensive experiments conducted
on the KITTI and Cityscapes dataset, our framework is shown to outperform the
state-of-the-art depth and motion estimation methods. Our code, dataset, and
models are available at this https URL .

本文提出了一种监督学习的、以单目摄像头为基础的端到端联合训练框架，该框架明确地建立了多个动态物体的 6-DoF 运动、自运动和景深，通过使用神经正向投影模块建立几何正确的投影流水线，在建模每个刚性物体的运动时突出了逆投影和正向投影之间的基本差异。通过使用任何现成的实例分割和光流模型，我们介绍了一种通用的自动注释方案来生成视频实例分割映射，这些映射将用作训练管道的输入。最后，我们使用 KITTI 和 Cityscapes 数据集进行了广泛的实验研究，证明了我们的框架优于现有的深度和运动估计方法。

通过实例感知的投影一致性学习动态场景下的单目深度

Learning Monocular Depth in Dynamic Scenes via Instance-Aware Projection  Consistency

We present an end-to-end joint training framework that explicitly models
6-DoF motion of multiple dynamic objects, ego-motion and depth in a monocular
camera setup without supervision. Our technical contributions are three-fold.
First, we propose a differentiable forward rigid projection module that plays a
key role in our instance-wise depth and motion learning. Second, we design an
instance-wise photometric and geometric consistency loss that effectively
decomposes background and moving object regions. Lastly, we introduce a new
auto-annotation scheme to produce video instance segmentation maps that will be
utilized as input to our training pipeline. These proposed elements are
validated in a detailed ablation study. Through extensive experiments conducted
on the KITTI dataset, our framework is shown to outperform the state-of-the-art
depth and motion estimation methods. Our code and dataset will be available at
this https URL

我们提出了一种全面的联合训练框架，该框架在单目相机设置中明确建模多个动态对象、自我运动和深度而不需要监督，并且具有高效的深度和运动估算，并通过实验证明了其优于现有深度和运动估算方法。