Scene Graph Generation (SGG) remains a challenging visual understanding task
due to its compositional property. Most previous works adopt a bottom-up,
two-stage or point-based, one-stage approach, which often suffers from high
time complexity or suboptimal designs. In this work, we propose a novel SGG
method to address the aforementioned issues, formulating the task as a
bipartite graph construction problem. To address the issues above, we create a
transformer-based end-to-end framework to generate the entity and entity-aware
predicate proposal set, and infer directed edges to form relation triplets.
Moreover, we design a graph assembling module to infer the connectivity of the
bipartite scene graph based on our entity-aware structure, enabling us to
generate the scene graph in an end-to-end manner. Based on bipartite graph
assembling paradigm, we further propose a new technical design to address the
efficacy of entity-aware modeling and optimization stability of graph
assembling. Equipped with the enhanced entity-aware design, our method achieves
optimal performance and time-complexity. Extensive experimental results show
that our design is able to achieve the state-of-the-art or comparable
performance on three challenging benchmarks, surpassing most of the existing
approaches and enjoying higher efficiency in inference. Code is available:
this https URL

通过使用双分图构建方法和终到终的框架，我们提出了一种新的场景图生成方法，有效地解决了组合属性所带来的挑战。实验证明，我们的方法在三个具有挑战性的基准测试中展现出了优秀的表现和高效率的推理能力，超过了现有方法的大部分，达到了最佳性能和时间复杂度。

SGTR+：使用 Transformer 进行端到端场景图生成

SGTR+: End-to-end Scene Graph Generation with Transformer

Scene Graph Generation (SGG) remains a challenging visual understanding task
due to its compositional property. Most previous works adopt a bottom-up
two-stage or a point-based one-stage approach, which often suffers from high
time complexity or sub-optimal designs. In this work, we propose a novel SGG
method to address the aforementioned issues, formulating the task as a
bipartite graph construction problem. To solve the problem, we develop a
transformer-based end-to-end framework that first generates the entity and
predicate proposal set, followed by inferring directed edges to form the
relation triplets. In particular, we develop a new entity-aware predicate
representation based on a structural predicate generator that leverages the
compositional property of relationships. Moreover, we design a graph assembling
module to infer the connectivity of the bipartite scene graph based on our
entity-aware structure, enabling us to generate the scene graph in an
end-to-end manner. Extensive experimental results show that our design is able
to achieve the state-of-the-art or comparable performance on two challenging
benchmarks, surpassing most of the existing approaches and enjoying higher
efficiency in inference. We hope our model can serve as a strong baseline for
the Transformer-based scene graph generation. Code is available:
this https URL

提出了一种基于 Transformer 的端到端框架，利用结构化谓语生成器开发了一种新的实体感知的谓语表示方法，设计了图组装模块以推断适应性的进行双部分场景图形的构建，并在两个具有挑战性的基准测试中取得了最先进或可比的性能，并提高了推理效率。