Scene graph generation (SGG) is to detect object pairs with their relations
in an image. Existing SGG approaches often use multi-stage pipelines to
decompose this task into object detection, relation graph construction, and
dense or dense-to-sparse relation prediction. Instead, from a perspective on
SGG as a direct set prediction, this paper presents a simple, sparse, and
unified framework, termed as Structured Sparse R-CNN. The key to our method is
a set of learnable triplet queries and a structured triplet detector which
could be jointly optimized from the training set in an end-to-end manner.
Specifically, the triplet queries encode the general prior for object pairs
with their relations, and provide an initial guess of scene graphs for
subsequent refinement. The triplet detector presents a cascaded architecture to
progressively refine the detected scene graphs with the customized dynamic
heads. In addition, to relieve the training difficulty of our method, we
propose a relaxed and enhanced training strategy based on knowledge
distillation from a Siamese Sparse R-CNN. We perform experiments on several
datasets: Visual Genome and Open Images V4/V6, and the results demonstrate that
our method achieves the state-of-the-art performance. In addition, we also
perform in-depth ablation studies to provide insights on our structured
modeling in triplet detector design and training strategies. The code and
models are made available at this https URL

本文提出一种名为 结构化稀疏 R-CNN 的框架，该框架从场景图生成（SGG）的角度出发，将其视为直接集合预测。文中使用可学习的三元组查询和结构化三元组检测器，并提出级联体系结构用于渐进地完善检测到的场景图。在多个数据集上进行的实验结果表明，该方法具有最先进的性能。