Panorama images have a much larger field-of-view thus naturally encode
enriched scene context information compared to standard perspective images,
which however is not well exploited in the previous scene understanding
methods. In this paper, we propose a novel method for panoramic 3D scene
understanding which recovers the 3D room layout and the shape, pose, position,
and semantic category for each object from a single full-view panorama image.
In order to fully utilize the rich context information, we design a novel graph
neural network based context model to predict the relationship among objects
and room layout, and a differentiable relationship-based optimization module to
optimize object arrangement with well-designed objective functions on-the-fly.
Realizing the existing data are either with incomplete ground truth or
overly-simplified scene, we present a new synthetic dataset with good diversity
in room layout and furniture placement, and realistic image quality for total
panoramic 3D scene understanding. Experiments demonstrate that our method
outperforms existing methods on panoramic scene understanding in terms of both
geometry accuracy and object arrangement. Code is available at
this https URL

本研究提出了一种基于全景图像的新颖的三维场景理解方法，通过设计基于图神经网络的上下文模型来预测物体之间的关系和房间布局，以及可微分的基于关系的优化模块来优化物体排列，实现对全景三维场景的理解， 并制备了具有多样化房间布局和家具摆放的真实感合成数据集，在几何精度和目标布置方面优于现有方法。