Dynamic scene graph generation (SGG) from videos requires not only
comprehensive understanding of objects across the scenes that are prone to
temporal fluctuations but also a model the temporal motions and interactions
with different objects. Moreover, the long-tailed distribution of visual
relationships is the crucial bottleneck of most dynamic SGG methods, since most
of them focus on capturing spatio-temporal context using complex architectures,
which leads to the generation of biased scene graphs. To address these
challenges, we propose FloCoDe: Flow-aware temporal consistency and Correlation
Debiasing with uncertainty attenuation for unbiased dynamic scene graphs.
FloCoDe employs feature warping using flow to detect temporally consistent
objects across the frames. In addition, it uses correlation debiasing to learn
the unbiased relation representation for long-tailed classes. Moreover, to
attenuate the predictive uncertainties, it uses a mixture of sigmoidal
cross-entropy loss and contrastive loss to incorporate label correlations to
identify the commonly co-occurring relations and help debias the long-tailed
ones. Extensive experimental evaluation shows a performance gain as high as
4.1% showing the superiority of generating more unbiased scene graphs.

FloCoDe 通过使用流进行特征扭曲以检测帧间时间一致的对象，并使用相关性去偏学习长尾类别的无偏关系表示，通过混合 sigmoid 交叉熵损失和对比损失以缓解预测不确定性，实现生成更加无偏的场景图，性能提升高达 4.1%。

视频中无偏场景图生成的相关性去偏

Correlation Debiasing for Unbiased Scene Graph Generation in Videos

The task of dynamic scene graph generation (SGG) from videos is complicated
and challenging due to the inherent dynamics of a scene, temporal fluctuation
of model predictions, and the long-tailed distribution of the visual
relationships in addition to the already existing challenges in image-based
SGG. Existing methods for dynamic SGG have primarily focused on capturing
spatio-temporal context using complex architectures without addressing the
challenges mentioned above, especially the long-tailed distribution of
relationships. This often leads to the generation of biased scene graphs. To
address these challenges, we introduce a new framework called TEMPURA: TEmporal
consistency and Memory Prototype guided UnceRtainty Attenuation for unbiased
dynamic SGG. TEMPURA employs object-level temporal consistencies via
transformer-based sequence modeling, learns to synthesize unbiased relationship
representations using memory-guided training, and attenuates the predictive
uncertainty of visual relations using a Gaussian Mixture Model (GMM). Extensive
experiments demonstrate that our method achieves significant (up to 10% in some
cases) performance gain over existing methods highlighting its superiority in
generating more unbiased scene graphs.

该研究提出了一个名为 TEMPURA 的框架，通过基于 Transformer 的序列建模，学习合成无偏差的关系表示，并使用高斯混合模型来减少视觉关系的预测不确定性，从而解决视频动态场景的场景图生成中存在的多种问题。该方法在生成更加无偏的场景图方面具有显著的优势。