Visual scene understanding is an important capability that enables robots to
purposefully act in their environment. In this paper, we propose a novel
approach to object-class segmentation from multiple RGB-D views using deep
learning. We train a deep neural network to predict object-class semantics that
is consistent from several view points in a semi-supervised way. At test time,
the semantics predictions of our network can be fused more consistently in
semantic keyframe maps than predictions of a network trained on individual
views. We base our network architecture on a recent single-view deep learning
approach to RGB and depth fusion for semantic object-class segmentation and
enhance it with multi-scale loss minimization. We obtain the camera trajectory
using RGB-D SLAM and warp the predictions of RGB-D images into ground-truth
annotated frames in order to enforce multi-view consistency during training. At
test time, predictions from multiple views are fused into keyframes. We propose
and analyze several methods for enforcing multi-view consistency during
training and testing. We evaluate the benefit of multi-view consistency
training and demonstrate that pooling of deep features and fusion over multiple
views outperforms single-view baselines on the NYUDv2 benchmark for semantic
segmentation. Our end-to-end trained network achieves state-of-the-art
performance on the NYUDv2 dataset in single-view segmentation as well as
multi-view semantic fusion.

本文提出一种从多个 RGB-D 视图中进行对象类语义分割的新方法，使用深度学习训练神经网络进行预测，同时借助于 RGB-D 的 SLAM 获取摄像机轨迹，实现多视图一致性训练和测试。实验结果表明，与单视图基准线相比，深度特征池化和多视角融合可以在 NYUDv2 数据集上实现单视图语义分割和多视图语义融合的最新性能。

使用 RGB-D 相机的多视角深度学习实现一致的语义映射

Multi-View Deep Learning for Consistent Semantic Mapping with RGB-D  Cameras

We present an open-source, real-time implementation of SemanticPaint, a
system for geometric reconstruction, object-class segmentation and learning of
3D scenes. Using our system, a user can walk into a room wearing a depth camera
and a virtual reality headset, and both densely reconstruct the 3D scene and
interactively segment the environment into object classes such as 'chair',
'floor' and 'table'. The user interacts physically with the real-world scene,
touching objects and using voice commands to assign them appropriate labels.
These user-generated labels are leveraged by an online random forest-based
machine learning algorithm, which is used to predict labels for previously
unseen parts of the scene. The entire pipeline runs in real time, and the user
stays 'in the loop' throughout the process, receiving immediate feedback about
the progress of the labelling and interacting with the scene as necessary to
refine the predicted segmentation.

本文介绍了 SemanticPaint 的实时开源实现，它能够实现几何重建、对象类别分割和 3D 场景学习；用户可以在佩戴深度相机和虚拟现实头戴设备的情况下，与真实场景进行物理交互并为场景中的对象分配标签，使用在线随机森林机器学习算法对先前未见过的场景部分进行预测，整个过程在实时运行中实现。