Scene graph generation (SGG) and human-object interaction (HOI) detection are
two important visual tasks aiming at localising and recognising relationships
between objects, and interactions between humans and objects, respectively.
Prevailing works treat these tasks as distinct tasks, leading to the
development of task-specific models tailored to individual datasets. However,
we posit that the presence of visual relationships can furnish crucial
contextual and intricate relational cues that significantly augment the
inference of human-object interactions. This motivates us to think if there is
a natural intrinsic relationship between the two tasks, where scene graphs can
serve as a source for inferring human-object interactions. In light of this, we
introduce SG2HOI+, a unified one-step model based on the Transformer
architecture. Our approach employs two interactive hierarchical Transformers to
seamlessly unify the tasks of SGG and HOI detection. Concretely, we initiate a
relation Transformer tasked with generating relation triples from a suite of
visual features. Subsequently, we employ another transformer-based decoder to
predict human-object interactions based on the generated relation triples. A
comprehensive series of experiments conducted across established benchmark
datasets including Visual Genome, V-COCO, and HICO-DET demonstrates the
compelling performance of our SG2HOI+ model in comparison to prevalent
one-stage SGG models. Remarkably, our approach achieves competitive performance
when compared to state-of-the-art HOI methods. Additionally, we observe that
our SG2HOI+ jointly trained on both SGG and HOI tasks in an end-to-end manner
yields substantial improvements for both tasks compared to individualized
training paradigms.

以 Transformer 架构为基础的统一一步模型 SG2HOI + 同时实现了场景图生成和人物对象互动检测，在视觉特征的基础上生成关系三元组，并基于此预测人物对象互动，实现了显著的性能提升，超过了现有单阶段场景图生成模型和最先进的人物对象互动方法。