Multi-scale deep CNNs have been used successfully for problems mapping each
pixel to a label, such as depth estimation and semantic segmentation. It has
also been shown that such architectures are reusable and can be used for
multiple tasks. These networks are typically trained independently for each
task by varying the output layer(s) and training objective. In this work we
present a new model for simultaneous depth estimation and semantic segmentation
from a single RGB image. Our approach demonstrates the feasibility of training
parts of the model for each task and then fine tuning the full, combined model
on both tasks simultaneously using a single loss function. Furthermore we
couple the deep CNN with fully connected CRF, which captures the contextual
relationships and interactions between the semantic and depth cues improving
the accuracy of the final results. The proposed model is trained and evaluated
on NYUDepth V2 dataset outperforming the state of the art methods on semantic
segmentation and achieving comparable results on the task of depth estimation.

本研究采用多尺度卷积神经网络，提出一种单一 RGB 图像用于深度估计和语义分割的深度学习模型，该模型采用单一损失函数进行训练，同时使用完全连接的 CRF 捕捉语义和深度线索之间的上下文关系和交互作用，实现更精确的结果。在 NYUDepth V2 数据集上进行实验，该模型在语义分割方面表现优于最先进方法，在深度估计任务上实现可比较的结果。