This paper addresses the problem of geometric scene parsing, i.e.
simultaneously labeling geometric surfaces (e.g. sky, ground and vertical
plane) and determining the interaction relations (e.g. layering, supporting,
siding and affinity) between main regions. This problem is more challenging
than the traditional semantic scene labeling, as recovering geometric
structures necessarily requires the rich and diverse contextual information. To
achieve these goals, we propose a novel recurrent neural network model, named
Hierarchical Long Short-Term Memory (H-LSTM). It contains two coupled
sub-networks: the Pixel LSTM (P-LSTM) and the Multi-scale Super-pixel LSTM
(MS-LSTM) for handling the surface labeling and relation prediction,
respectively. The two sub-networks provide complementary information to each
other to exploit hierarchical scene contexts, and they are jointly optimized
for boosting the performance. Our extensive experiments show that our model is
capable of parsing scene geometric structures and outperforming several
state-of-the-art methods by large margins. In addition, we show promising 3D
reconstruction results from the still images based on the geometric parsing.

本研究提出了一种名为 Hierarchical Long Short-Term Memory（H-LSTM）的循环神经网络模型，它包含两个耦合子网络，即 Pixel LSTM 和 Multi-scale Super-pixel LSTM，用于处理表面标记和关系预测，分别提供补充信息以利用分层场景上下文，联合优化以提高性能，并能在几何场景中解析场景几何结构，并且具有优秀的 3D 重建结果。