Panoptic Scene Graph has recently been proposed for comprehensive scene
understanding. However, previous works adopt a fully-supervised learning
manner, requiring large amounts of pixel-wise densely-annotated data, which is
always tedious and expensive to obtain. To address this limitation, we study a
new problem of Panoptic Scene Graph Generation from Purely Textual Descriptions
(Caption-to-PSG). The key idea is to leverage the large collection of free
image-caption data on the Web alone to generate panoptic scene graphs. The
problem is very challenging for three constraints: 1) no location priors; 2) no
explicit links between visual regions and textual entities; and 3) no
pre-defined concept sets. To tackle this problem, we propose a new framework
TextPSG consisting of four modules, i.e., a region grouper, an entity grounder,
a segment merger, and a label generator, with several novel techniques. The
region grouper first groups image pixels into different segments and the entity
grounder then aligns visual segments with language entities based on the
textual description of the segment being referred to. The grounding results can
thus serve as pseudo labels enabling the segment merger to learn the segment
similarity as well as guiding the label generator to learn object semantics and
relation predicates, resulting in a fine-grained structured scene
understanding. Our framework is effective, significantly outperforming the
baselines and achieving strong out-of-distribution robustness. We perform
comprehensive ablation studies to corroborate the effectiveness of our design
choices and provide an in-depth analysis to highlight future directions. Our
code, data, and results are available on our project page:
this https URL

从纯文本描述中生成全景场景图，通过区域分组器、实体对齐器、段合并器和标签生成器构建无显式链接、无预定义概念集的场景理解框架，有效地提高了性能和鲁棒性。