Towards building comprehensive real-world visual perception systems, we
propose and study a new problem called panoptic scene graph generation (PVSG).
PVSG relates to the existing video scene graph generation (VidSGG) problem,
which focuses on temporal interactions between humans and objects grounded with
bounding boxes in videos. However, the limitation of bounding boxes in
detecting non-rigid objects and backgrounds often causes VidSGG to miss key
details crucial for comprehensive video understanding. In contrast, PVSG
requires nodes in scene graphs to be grounded by more precise, pixel-level
segmentation masks, which facilitate holistic scene understanding. To advance
research in this new area, we contribute the PVSG dataset, which consists of
400 videos (289 third-person + 111 egocentric videos) with a total of 150K
frames labeled with panoptic segmentation masks as well as fine, temporal scene
graphs. We also provide a variety of baseline methods and share useful design
practices for future work.

通过建立综合的真实世界视觉感知系统，我们提出并研究了一个称为全景场景图生成（PVSG）的新问题。PVSG 与现有的视频场景图生成（VidSGG）问题相关，后者侧重于视频中的人与物体之间的时间交互，并基于边界框进行实体识别。然而，边界框在检测非刚性物体和背景方面的局限性常常导致 VidSGG 遗漏关键细节，而 PVSG 则要求场景图中的节点由更精确的像素级分割掩码实体识别，以促进整体场景理解。为了推动这一新领域的研究，我们贡献了 PVSG 数据集，其中包含 400 个视频（289 个第三人称视频 + 111 个自我中心视频），包含 150K 帧用于全景分割掩码以及精细的时间场景图。我们还提供了各种基准方法，并分享了未来工作的有用设计实践。