We propose a computational framework to jointly parse a single RGB image and
reconstruct a holistic 3D configuration composed by a set of CAD models using a
stochastic grammar model. Specifically, we introduce a Holistic Scene Grammar
(HSG) to represent the 3D scene structure, which characterizes a joint
distribution over the functional and geometric space of indoor scenes. The
proposed HSG captures three essential and often latent dimensions of the indoor
scenes: i) latent human context, describing the affordance and the
functionality of a room arrangement, ii) geometric constraints over the scene
configurations, and iii) physical constraints that guarantee physically
plausible parsing and reconstruction. We solve this joint parsing and
reconstruction problem in an analysis-by-synthesis fashion, seeking to minimize
the differences between the input image and the rendered images generated by
our 3D representation, over the space of depth, surface normal, and object
segmentation map. The optimal configuration, represented by a parse graph, is
inferred using Markov chain Monte Carlo (MCMC), which efficiently traverses
through the non-differentiable solution space, jointly optimizing object
localization, 3D layout, and hidden human context. Experimental results
demonstrate that the proposed algorithm improves the generalization ability and
significantly outperforms prior methods on 3D layout estimation, 3D object
detection, and holistic scene understanding.

本文提出了一种使用随机语法模型联合解析单个 RGB 图像并重建由一组 CAD 模型组成的整体 3D 配置的计算框架，并引入了一个全局场景语法（HSG）表示 3D 场景结构，进而以分析 - 合成方式解决该联合分析和重建问题。