Dynamic scene graphs generated from video clips could help enhance the
semantic visual understanding in a wide range of challenging tasks such as
environmental perception, autonomous navigation, and task planning of
self-driving vehicles and mobile robots. In the process of temporal and spatial
modeling during dynamic scene graph generation, it is particularly intractable
to learn time-variant relations in dynamic scene graphs among frames. In this
paper, we propose a Time-variant Relation-aware TRansformer (TR$^2$), which
aims to model the temporal change of relations in dynamic scene graphs.
Explicitly, we leverage the difference of text embeddings of prompted sentences
about relation labels as the supervision signal for relations. In this way,
cross-modality feature guidance is realized for the learning of time-variant
relations. Implicitly, we design a relation feature fusion module with a
transformer and an additional message token that describes the difference
between adjacent frames. Extensive experiments on the Action Genome dataset
prove that our TR$^2$ can effectively model the time-variant relations. TR$^2$
significantly outperforms previous state-of-the-art methods under two different
settings by 2.1% and 2.6% respectively.

本研究提出了一种 Time-variant Relation-aware TRansformer（TR^2）模型，通过学习提示句子中关系标签的嵌入差异，实现了跨模态特征指导，设计了一个关系特征融合模块与转换器以及描述相邻帧之间差异的额外信息令牌，以有效建模动态场景图中的时变关系，实验证明其在 Action Genome 数据集上的表现优于之前最先进研究方法。

跨模态时变关系学习用于生成动态场景图

Cross-Modality Time-Variant Relation Learning for Generating Dynamic  Scene Graphs

Pretrained language models (PLMs) for data-to-text (D2T) generation can use
human-readable data labels such as column headings, keys, or relation names to
generalize to out-of-domain examples. However, the models are well-known in
producing semantically inaccurate outputs if these labels are ambiguous or
incomplete, which is often the case in D2T datasets. In this paper, we expose
this issue on the task of descibing a relation between two entities. For our
experiments, we collect a novel dataset for verbalizing a diverse set of 1,522
unique relations from three large-scale knowledge graphs (Wikidata, DBPedia,
YAGO). We find that although PLMs for D2T generation expectedly fail on unclear
cases, models trained with a large variety of relation labels are surprisingly
robust in verbalizing novel, unseen relations. We argue that using data with a
diverse set of clear and meaningful labels is key to training D2T generation
systems capable of generalizing to novel domains.

本文研究预训练语言模型在数据到文本生成中使用关系标签生成语义准确输出时的问题，并通过实验发现，使用多样化的关系标签数据可以提高模型在生成新领域的语义准确性。

利用预训练模型描述知识图谱中的关系

Mind the Labels: Describing Relations in Knowledge Graphs With Pretrained Models

We describe an inventory of semantic relations that are expressed by
prepositions. We define these relations by building on the word sense
disambiguation task for prepositions and propose a mapping from preposition
senses to the relation labels by collapsing semantically related senses across
prepositions.

该研究介绍了一个预先设定的语义关系清单，重点围绕介词表达的语义关系，同时运用了介词词义消歧的任务和将介词词义映射到关系标签的技术，通过合并语义相近的介词词义，进一步定义了这些关系。