While deep neural networks have led to human-level performance on computer vision tasks, they have yet to demonstrate similar gains for holistic scene understanding. In particular, 3D context has been shown to be an extremely important cue for scene understanding - yet very little research has been done on integrating context information with deep models. This paper presents an approach to embed 3D context into the topology of a neural network trained to perform holistic scene understanding. Given a depth image depicting a 3D scene, our network aligns the observed scene with a predefined 3D scene template, and then reasons about the existence and location of each object within the scene template. In doing so, our model recognizes multiple objects in a single forward pass of a 3D convolutional neural network, capturing both global scene and local object information simultaneously. To create training data for this 3D network, we generate partly hallucinated depth images which are rendered by replacing real objects with a repository of CAD models of the same object category. Extensive experiments demonstrate the effectiveness of our algorithm compared to the state-of-the-arts. Source code and data will be available.

本文提出通过将3D上下文信息嵌入到神经网络拓扑中以实现整体场景理解的方法，为此我们采用了一个神经网络对接收到的深度图像进行对齐并推断其中物体的存在和位置，以及使用部分虚构的深度图像生成数据集，所有实验结果都表明了本算法的有效性。

DeepContext：用于3D完整场景理解的上下文编码神经路径