Panoptic Scene Graph (PSG) is a challenging task in Scene Graph Generation
(SGG) that aims to create a more comprehensive scene graph representation using
panoptic segmentation instead of boxes. However, current PSG methods have
limited performance, which can hinder downstream task development. To improve
PSG methods, we conducted an in-depth analysis to identify the bottleneck of
the current PSG models, finding that inter-object pair-wise recall is a crucial
factor which was ignored by previous PSG methods. Based on this, we present a
novel framework: Pair then Relation (Pair-Net), which uses a Pair Proposal
Network (PPN) to learn and filter sparse pair-wise relationships between
subjects and objects. We also observed the sparse nature of object pairs and
used this insight to design a lightweight Matrix Learner within the PPN.
Through extensive ablation and analysis, our approach significantly improves
upon leveraging the strong segmenter baseline. Notably, our approach achieves
new state-of-the-art results on the PSG benchmark, with over 10% absolute gains
compared to PSGFormer. The code of this paper is publicly available at
this https URL

本文提出了一种基于 Pair Proposal Network (PPN) 的新型架构 Pair then Relation (Pair-Net)，利用矩阵学习器 (matrix learner) 来过滤稀疏的主语和宾语之间的关系，通过实验分析得到了新的最佳 PSG 表现结果，并超越了之前的 PSGFormer。