Spatio-temporal scene graphs represent interactions in a video by decomposing
scenes into individual objects and their pair-wise temporal relationships.
Long-term anticipation of the fine-grained pair-wise relationships between
objects is a challenging problem. To this end, we introduce the task of Scene
Graph Anticipation (SGA). We adapt state-of-the-art scene graph generation
methods as baselines to anticipate future pair-wise relationships between
objects and propose a novel approach SceneSayer. In SceneSayer, we leverage
object-centric representations of relationships to reason about the observed
video frames and model the evolution of relationships between objects. We take
a continuous time perspective and model the latent dynamics of the evolution of
object interactions using concepts of NeuralODE and NeuralSDE, respectively. We
infer representations of future relationships by solving an Ordinary
Differential Equation and a Stochastic Differential Equation, respectively.
Extensive experimentation on the Action Genome dataset validates the efficacy
of the proposed methods.

分析了视频中的时空场景图，提出了 SceneSayer 方法，通过对观察到的视频帧进行推理，模拟对象之间关系的演化，并使用神经常微分方程和神经随机微分方程来预测对象之间的未来关系。在 Action Genome 数据集上进行了大量实验验证了该方法的效果。

走向场景图预测

Towards Scene Graph Anticipation

Dynamic scene graphs generated from video clips could help enhance the
semantic visual understanding in a wide range of challenging tasks such as
environmental perception, autonomous navigation, and task planning of
self-driving vehicles and mobile robots. In the process of temporal and spatial
modeling during dynamic scene graph generation, it is particularly intractable
to learn time-variant relations in dynamic scene graphs among frames. In this
paper, we propose a Time-variant Relation-aware TRansformer (TR$^2$), which
aims to model the temporal change of relations in dynamic scene graphs.
Explicitly, we leverage the difference of text embeddings of prompted sentences
about relation labels as the supervision signal for relations. In this way,
cross-modality feature guidance is realized for the learning of time-variant
relations. Implicitly, we design a relation feature fusion module with a
transformer and an additional message token that describes the difference
between adjacent frames. Extensive experiments on the Action Genome dataset
prove that our TR$^2$ can effectively model the time-variant relations. TR$^2$
significantly outperforms previous state-of-the-art methods under two different
settings by 2.1% and 2.6% respectively.

本研究提出了一种 Time-variant Relation-aware TRansformer（TR^2）模型，通过学习提示句子中关系标签的嵌入差异，实现了跨模态特征指导，设计了一个关系特征融合模块与转换器以及描述相邻帧之间差异的额外信息令牌，以有效建模动态场景图中的时变关系，实验证明其在 Action Genome 数据集上的表现优于之前最先进研究方法。