Panoramic image enables deeper understanding and more holistic perception of
$360^\circ$ surrounding environment, which can naturally encode enriched scene
context information compared to standard perspective image. Previous work has
made lots of effort to solve the scene understanding task in a bottom-up form,
thus each sub-task is processed separately and few correlations are explored in
this procedure. In this paper, we propose a novel method using depth prior for
holistic indoor scene understanding which recovers the objects' shapes,
oriented bounding boxes and the 3D room layout simultaneously from a single
panorama. In order to fully utilize the rich context information, we design a
transformer-based context module to predict the representation and relationship
among each component of the scene. In addition, we introduce a real-world
dataset for scene understanding, including photo-realistic panoramas,
high-fidelity depth images, accurately annotated room layouts, and oriented
object bounding boxes and shapes. Experiments on the synthetic and real-world
datasets demonstrate that our method outperforms previous panoramic scene
understanding methods in terms of both layout estimation and 3D object
detection.

本文提出了一种使用深度先验的方法，从单个全景图中同时复原物体形状、定向边界框和三维房间布局，为了充分利用丰富的上下文信息，设计了一个基于 transformer 的上下文模块来预测场景各组成部分之间的表示和关系。实验表明，我们的方法在布局估计和三维物体检测方面优于先前的全景场景理解方法。同时，本文还引入了一个包括照片逼真的全景图、高保真深度图、精确注释的房间布局和定向物体边界框和形状的真实世界数据集。