Video scene graph generation (VidSGG) aims to identify objects in visual
scenes and infer their relationships for a given video. It requires not only a
comprehensive understanding of each object scattered on the whole scene but
also a deep dive into their temporal motions and interactions. Inherently,
object pairs and their relationships enjoy spatial co-occurrence correlations
within each image and temporal consistency/transition correlations across
different images, which can serve as prior knowledge to facilitate VidSGG model
learning and inference. In this work, we propose a spatial-temporal
knowledge-embedded transformer (STKET) that incorporates the prior
spatial-temporal knowledge into the multi-head cross-attention mechanism to
learn more representative relationship representations. Specifically, we first
learn spatial co-occurrence and temporal transition correlations in a
statistical manner. Then, we design spatial and temporal knowledge-embedded
layers that introduce the multi-head cross-attention mechanism to fully explore
the interaction between visual representation and the knowledge to generate
spatial- and temporal-embedded representations, respectively. Finally, we
aggregate these representations for each subject-object pair to predict the
final semantic labels and their relationships. Extensive experiments show that
STKET outperforms current competing algorithms by a large margin, e.g.,
improving the mR@50 by 8.1%, 4.7%, and 2.1% on different settings over current
algorithms.

提出了一种将空间 - 时间知识嵌入到多头交叉注意力机制中的空间 - 时间知识嵌入变压器（STKET）模型，用于生成视频场景图并预测语义标签及其关系。经过广泛实验表明，STKET 在不同设置下相对于当前算法的 mR@50 提升了 8.1％，4.7％和 2.1％。

视频场景图生成的时空知识嵌入 Transformer

Spatial-Temporal Knowledge-Embedded Transformer for Video Scene Graph  Generation

Knowledge-enhanced pre-trained models for language representation have been
shown to be more effective in knowledge base construction tasks (i.e.,~relation
extraction) than language models such as BERT. These knowledge-enhanced
language models incorporate knowledge into pre-training to generate
representations of entities or relationships. However, existing methods
typically represent each entity with a separate embedding. As a result, these
methods struggle to represent out-of-vocabulary entities and a large amount of
parameters, on top of their underlying token models (i.e.,~the transformer),
must be used and the number of entities that can be handled is limited in
practice due to memory constraints. Moreover, existing models still struggle to
represent entities and relationships simultaneously. To address these problems,
we propose a new pre-trained model that learns representations of both entities
and relationships from token spans and span pairs in the text respectively. By
encoding spans efficiently with span modules, our model can represent both
entities and their relationships but requires fewer parameters than existing
models. We pre-trained our model with the knowledge graph extracted from
Wikipedia and test it on a broad range of supervised and unsupervised
information extraction tasks. Results show that our model learns better
representations for both entities and relationships than baselines, while in
supervised settings, fine-tuning our model outperforms RoBERTa consistently and
achieves competitive results on information extraction tasks.

我们提出了一种新的预训练模型，利用基于维基百科的知识图谱实现了在实体和关系上的双重表示，以及利用跨度模块进行有效编码，相较于现有模型使用更少的参数。实验结果表明，相比 RoBERTa 模型，我们模型在信息抽取任务上表现更好，尤其在监督学习任务中达到了很高的竞争水平。