This paper investigates the problem of scene graph generation in videos with
the aim of capturing semantic relations between subjects and objects in the
form of $\langle$subject, predicate, object$\rangle$ triplets. Recognizing the
predicate between subject and object pairs is imbalanced and multi-label in
nature, ranging from ubiquitous interactions such as spatial relationships (\eg
\emph{in front of}) to rare interactions such as \emph{twisting}. In
widely-used benchmarks such as Action Genome and VidOR, the imbalance ratio
between the most and least frequent predicates reaches 3,218 and 3,408,
respectively, surpassing even benchmarks specifically designed for long-tailed
recognition. Due to the long-tailed distributions and label co-occurrences,
recent state-of-the-art methods predominantly focus on the most frequently
occurring predicate classes, ignoring those in the long tail. In this paper, we
analyze the limitations of current approaches for scene graph generation in
videos and identify a one-to-one correspondence between predicate frequency and
recall performance. To make the step towards unbiased scene graph generation in
videos, we introduce a multi-label meta-learning framework to deal with the
biased predicate distribution. Our meta-learning framework learns a meta-weight
network for each training sample over all possible label losses. We evaluate
our approach on the Action Genome and VidOR benchmarks by building upon two
current state-of-the-art methods for each benchmark. The experiments
demonstrate that the multi-label meta-weight network improves the performance
for predicates in the long tail without compromising performance for head
classes, resulting in better overall performance and favorable
generalizability. Code: https://github.com/shanshuo/ML-MWN.

本文研究了视频场景图生成中主题和对象之间语义关系的问题，提出了一种多标签元学习框架以处理偏斜的谓词分布，改进了现有方法并在 Action Genome 和 VidOR 基准测试中表现优异。