Scene Graph Generation (SGG) aims to identify entities and predict the
relationship triplets \textit{\textless subject, predicate, object\textgreater
} in visual scenes. Given the prevalence of large visual variations of
subject-object pairs even in the same predicate, it can be quite challenging to
model and refine predicate representations directly across such pairs, which is
however a common strategy adopted by most existing SGG methods. We observe that
visual variations within the identical triplet are relatively small and certain
relation cues are shared in the same type of triplet, which can potentially
facilitate the relation learning in SGG. Moreover, for the long-tail problem
widely studied in SGG task, it is also crucial to deal with the limited types
and quantity of triplets in tail predicates. Accordingly, in this paper, we
propose a Dual-granularity Relation Modeling (DRM) network to leverage
fine-grained triplet cues besides the coarse-grained predicate ones. DRM
utilizes contexts and semantics of predicate and triplet with Dual-granularity
Constraints, generating compact and balanced representations from two
perspectives to facilitate relation recognition. Furthermore, a
Dual-granularity Knowledge Transfer (DKT) strategy is introduced to transfer
variation from head predicates/triplets to tail ones, aiming to enrich the
pattern diversity of tail classes to alleviate the long-tail problem. Extensive
experiments demonstrate the effectiveness of our method, which establishes new
state-of-the-art performance on Visual Genome, Open Image, and GQA datasets.
Our code is available at https://github.com/jkli1998/DRM

本文提出了一种双粒度关系建模（Dual-granularity Relation Modeling）网络，利用粗粒度和细粒度的三元组线索来促进关系识别，并引入了双粒度知识传递（Dual-granularity Knowledge Transfer）策略来解决长尾问题，实验证明了该方法的有效性。