Scene understanding is a prerequisite to many high level tasks for any
automated intelligent machine operating in real world environments. Recent
attempts with supervised learning have shown promise in this direction but also
highlighted the need for enormous quantity of supervised data --- performance
increases in proportion to the amount of data used. However, this quickly
becomes prohibitive when considering the manual labour needed to collect such
data. In this work, we focus our attention on depth based semantic per-pixel
labelling as a scene understanding problem and show the potential of computer
graphics to generate virtually unlimited labelled data from synthetic 3D
scenes. By carefully synthesizing training data with appropriate noise models
we show comparable performance to state-of-the-art RGBD systems on NYUv2
dataset despite using only depth data as input and set a benchmark on
depth-based segmentation on SUN RGB-D dataset. Additionally, we offer a route
to generating synthesized frame or video data, and understanding of different
factors influencing performance gains.

本文研究场景理解问题，通过使用计算机图形学生成合成 3D 场景，以解决手动收集大量数据问题，从而避免了收集数据的麻烦和昂贵成本。通过使用深度数据作为输入仅使用 RGB-D 系统的深度数据，我们在 NYUv2 数据集上的表现相当于最先进的 RGBD 系统，并在 SUN RGB-D 数据集上设置了基准。文章还探讨了生成合成图像或视频数据，以及分析影响性能增益的不同因素。

使用合成数据理解现实世界室内场景的 SceneNet

SceneNet: Understanding Real World Indoor Scenes With Synthetic Data

We propose a new CNN-CRF end-to-end learning framework, which is based on
joint stochastic optimization with respect to both Convolutional Neural Network
(CNN) and Conditional Random Field (CRF) parameters. While stochastic gradient
descent is a standard technique for CNN training, it was not used for joint
models so far. We show that our learning method is (i) general, i.e. it applies
to arbitrary CNN and CRF architectures and potential functions; (ii) scalable,
i.e. it has a low memory footprint and straightforwardly parallelizes on GPUs;
(iii) easy in implementation. Additionally, the unified CNN-CRF optimization
approach simplifies a potential hardware implementation. We empirically
evaluate our method on the task of semantic labeling of body parts in depth
images and show that it compares favorably to competing techniques.

我们提出了一种基于 CNN 和 CRF 参数的联合随机优化的新的 CNN-CRF 端到端学习框架，经实验表明这种方法在深度图像身体部件的语义标记上比竞争技术表现更好。

使用随机优化方法联合训练通用 CNN-CRF 模型

Joint Training of Generic CNN-CRF Models with Stochastic Optimization

In this paper we address three different computer vision tasks using a single
basic architecture: depth prediction, surface normal estimation, and semantic
labeling. We use a multiscale convolutional network that is able to adapt
easily to each task using only small modifications, regressing from the input
image to the output map directly. Our method progressively refines predictions
using a sequence of scales, and captures many image details without any
superpixels or low-level segmentation. We achieve state-of-the-art performance
on benchmarks for all three tasks.

本文提出了一个利用多尺度卷积神经网络实现计算机视觉中深度预测、表面法线估计和语义标签等任务的方法，并在各项任务的基准测试中实现了最优表现。

使用共同的多尺度卷积结构预测深度，表面法线和语义标签

Predicting Depth, Surface Normals and Semantic Labels with a Common  Multi-Scale Convolutional Architecture

We frame the task of predicting a semantic labeling as a sparse
reconstruction procedure that applies a target-specific learned transfer
function to a generic deep sparse code representation of an image. This
strategy partitions training into two distinct stages. First, in an
unsupervised manner, we learn a set of generic dictionaries optimized for
sparse coding of image patches. We train a multilayer representation via
recursive sparse dictionary learning on pooled codes output by earlier layers.
Second, we encode all training images with the generic dictionaries and learn a
transfer function that optimizes reconstruction of patches extracted from
annotated ground-truth given the sparse codes of their corresponding image
patches. At test time, we encode a novel image using the generic dictionaries
and then reconstruct using the transfer function. The output reconstruction is
a semantic labeling of the test image.
Applying this strategy to the task of contour detection, we demonstrate
performance competitive with state-of-the-art systems. Unlike almost all prior
work, our approach obviates the need for any form of hand-designed features or
filters. To illustrate general applicability, we also show initial results on
semantic part labeling of human faces.
The effectiveness of our approach opens new avenues for research on deep
sparse representations. Our classifiers utilize this representation in a novel
manner. Rather than acting on nodes in the deepest layer, they attach to nodes
along a slice through multiple layers of the network in order to make
predictions about local patches. Our flexible combination of a generatively
learned sparse representation with discriminatively trained transfer
classifiers extends the notion of sparse reconstruction to encompass arbitrary
semantic labeling tasks.

通过学习稀疏编码图像的通用稀疏编码字典和针对特定标签的转换函数，我们的方法可以将测试图像进行语义分割。我们在轮廓检测任务上展示了与当前最先进系统相当的性能，并展示了在人脸语义分割任务中的初步结果。