Scene graph generation aims to capture detailed spatial and semantic
relationships between objects in an image, which is challenging due to
incomplete labelling, long-tailed relationship categories, and relational
semantic overlap. Existing Transformer-based methods either employ distinct
queries for objects and predicates or utilize holistic queries for relation
triplets and hence often suffer from limited capacity in learning low-frequency
relationships. In this paper, we present a new Transformer-based method, called
DSGG, that views scene graph detection as a direct graph prediction problem
based on a unique set of graph-aware queries. In particular, each graph-aware
query encodes a compact representation of both the node and all of its
relations in the graph, acquired through the utilization of a relaxed sub-graph
matching during the training process. Moreover, to address the problem of
relational semantic overlap, we utilize a strategy for relation distillation,
aiming to efficiently learn multiple instances of semantic relationships.
Extensive experiments on the VG and the PSG datasets show that our model
achieves state-of-the-art results, showing a significant improvement of 3.5\%
and 6.7\% in mR@50 and mR@100 for the scene-graph generation task and achieves
an even more substantial improvement of 8.5\% and 10.3\% in mR@50 and mR@100
for the panoptic scene graph generation task. Code is available at
https://github.com/zeeshanhayder/DSGG.

本文介绍了一种新的基于 Transformer 的方法，称为 DSGG，将场景图检测视为基于一组独特的图感知查询的直接图预测问题，并采用放松子图匹配的方式获取图节点及其关系的紧凑表示，以及通过关系蒸馏策略来解决关系语义重叠问题。实验结果表明，该模型在场景图生成任务上取得了最先进的结果，在 mR@50 和 mR@100 方面分别取得了 3.5％和 6.7％的显著改进，在全景场景图生成任务上甚至取得了 8.5％和 10.3％的更大改进。