We are living in a three-dimensional space while moving forward through a
fourth dimension: time. To allow artificial intelligence to develop a
comprehensive understanding of such a 4D environment, we introduce 4D Panoptic
Scene Graph (PSG-4D), a new representation that bridges the raw visual data
perceived in a dynamic 4D world and high-level visual understanding.
Specifically, PSG-4D abstracts rich 4D sensory data into nodes, which represent
entities with precise location and status information, and edges, which capture
the temporal relations. To facilitate research in this new area, we build a
richly annotated PSG-4D dataset consisting of 3K RGB-D videos with a total of
1M frames, each of which is labeled with 4D panoptic segmentation masks as well
as fine-grained, dynamic scene graphs. To solve PSG-4D, we propose PSG4DFormer,
a Transformer-based model that can predict panoptic segmentation masks, track
masks along the time axis, and generate the corresponding scene graphs via a
relation component. Extensive experiments on the new dataset show that our
method can serve as a strong baseline for future research on PSG-4D. In the
end, we provide a real-world application example to demonstrate how we can
achieve dynamic scene understanding by integrating a large language model into
our PSG-4D system.

在本文中，我们介绍了一种名为 4D Panoptic Scene Graph（PSG-4D）的新表示方法，用于在动态的四维世界中桥接原始视觉数据和高层次视觉理解。我们构建了一个富注释的 PSG-4D 数据集，并提出了 PSG4DFormer 模型，该模型可以预测全景分割掩码、跟踪掩码并通过关系组件生成相应的场景图。我们的方法在新数据集上进行了广泛实验，表明它可以作为 PSG-4D 未来研究的一个强有力的基准。最后，我们提供了一个真实世界的应用示例，以展示如何通过将大型语言模型整合到我们的 PSG-4D 系统中来实现动态场景理解。