Self-supervised learning is showing great promise for monocular depth
estimation, using geometry as the only source of supervision. Depth networks
are indeed capable of learning representations that relate visual appearance to
3D properties by implicitly leveraging category-level patterns. In this work we
investigate how to leverage more directly this semantic structure to guide
geometric representation learning, while remaining in the self-supervised
regime. Instead of using semantic labels and proxy losses in a multi-task
approach, we propose a new architecture leveraging fixed pretrained semantic
segmentation networks to guide self-supervised representation learning via
pixel-adaptive convolutions. Furthermore, we propose a two-stage training
process to overcome a common semantic bias on dynamic objects via resampling.
Our method improves upon the state of the art for self-supervised monocular
depth prediction over all pixels, fine-grained details, and per semantic
categories.

本文研究自监督学习中如何直接运用语义结构来指导几何表示的学习，提出了一种新的基于预训练语义分割网络和像素自适应卷积的架构，并使用两阶段训练过程来克服动态对象上的常见语义偏差。结果表明该方法在所有像素、细节和语义类别上都优于现有方法，实现了自监督单目深度预测的最新水平。