D semantic scene graphs are a powerful holistic representation as they
describe the individual objects and depict the relation between them. They are
compact high-level graphs that enable many tasks requiring scene reasoning. In
real-world settings, existing 3D estimation methods produce robust predictions
that mostly rely on dense inputs. In this work, we propose a real-time
framework that incrementally builds a consistent 3D semantic scene graph of a
scene given an RGB image sequence. Our method consists of a novel incremental
entity estimation pipeline and a scene graph prediction network. The proposed
pipeline simultaneously reconstructs a sparse point map and fuses entity
estimation from the input images. The proposed network estimates 3D semantic
scene graphs with iterative message passing using multi-view and geometric
features extracted from the scene entities. Extensive experiments on the 3RScan
dataset show the effectiveness of the proposed method in this challenging task,
outperforming state-of-the-art approaches.

本论文提出一种实时框架，使用 RGB 图像序列增量构建一致的三维语义场景图，包括新颖的增量实体估计流水线和场景图预测网络，使用多视角和几何特征，通过迭代信息传递估计三维语义场景图。经过在 3RScan 数据集上大量实验证明，该方法在这项挑战性任务中表现良好，优于现有的最先进方法。

RGB 序列递增式 3D 语义场景图预测

Incremental 3D Semantic Scene Graph Prediction from RGB Sequences

Despite recent advancements in single-domain or single-object image
generation, it is still challenging to generate complex scenes containing
diverse, multiple objects and their interactions. Scene graphs, composed of
nodes as objects and directed-edges as relationships among objects, offer an
alternative representation of a scene that is more semantically grounded than
images. We hypothesize that a generative model for scene graphs might be able
to learn the underlying semantic structure of real-world scenes more
effectively than images, and hence, generate realistic novel scenes in the form
of scene graphs. In this work, we explore a new task for the unconditional
generation of semantic scene graphs. We develop a deep auto-regressive model
called SceneGraphGen which can directly learn the probability distribution over
labelled and directed graphs using a hierarchical recurrent architecture. The
model takes a seed object as input and generates a scene graph in a sequence of
steps, each step generating an object node, followed by a sequence of
relationship edges connecting to the previous nodes. We show that the scene
graphs generated by SceneGraphGen are diverse and follow the semantic patterns
of real-world scenes. Additionally, we demonstrate the application of the
generated graphs in image synthesis, anomaly detection and scene graph
completion.

本文提出了一个用于无条件生成场景图的生成模型 SceneGraphGen，通过层级循环架构直接学习带标签和有向图的概率分布，生成的场景图多样且遵循真实场景的语义模式，同时还证明了生成的图在图像合成、异常检测和场景图完整性方面的应用。

无条件场景图生成

Unconditional Scene Graph Generation

Scene graphs are a compact and explicit representation successfully used in a
variety of 2D scene understanding tasks. This work proposes a method to
incrementally build up semantic scene graphs from a 3D environment given a
sequence of RGB-D frames. To this end, we aggregate PointNet features from
primitive scene components by means of a graph neural network. We also propose
a novel attention mechanism well suited for partial and missing graph data
present in such an incremental reconstruction scenario. Although our proposed
method is designed to run on submaps of the scene, we show it also transfers to
entire 3D scenes. Experiments show that our approach outperforms 3D scene graph
prediction methods by a large margin and its accuracy is on par with other 3D
semantic and panoptic segmentation methods while running at 35 Hz.

本文提出了一种利用图神经网络从三维环境中给定的一系列 RGB-D 帧逐步建立语义场景图的方法，并提出了一种适用于部分和缺失图数据的新型注意机制。实验表明，我们的方法在某些方面表现优于三维场景图预测方法，并且精度与其他三维语义方法和全景分割方法相当，同时运行速度达到每秒 35 帧。