Transformers flexibly operate over sets of real-valued vectors representing
task-specific entities and their attributes, where each vector might encode one
word-piece token and its position in a sequence, or some piece of information
that carries no position at all. But as set processors, transformers are at a
disadvantage in reasoning over more general graph-structured data where nodes
represent entities and edges represent relations between entities. To address
this shortcoming, we generalize transformer attention to consider and update
edge vectors in each transformer layer. We evaluate this relational transformer
on a diverse array of graph-structured tasks, including the large and
challenging CLRS Algorithmic Reasoning Benchmark. There, it dramatically
outperforms state-of-the-art graph neural networks expressly designed to reason
over graph-structured data. Our analysis demonstrates that these gains are
attributable to relational attention's inherent ability to leverage the greater
expressivity of graphs over sets.

本文描述了如何使用关系注意力处理图数据，以及在各种基准测试中比图神经网络做得更好的关系变压器。

基于关系注意力的图结构任务通用 Transformer

Relational Attention: Generalizing Transformers for Graph-Structured Tasks

This paper presents a novel framework to integrate both semantic and instance
contexts for panoptic segmentation. In existing works, it is common to use a
shared backbone to extract features for both things (countable classes such as
vehicles) and stuff (uncountable classes such as roads). This, however, fails
to capture the rich relations among them, which can be utilized to enhance
visual understanding and segmentation performance. To address this shortcoming,
we propose a novel Panoptic, Instance, and Semantic Relations (PISR) module to
exploit such contexts. First, we generate panoptic encodings to summarize key
features of the semantic classes and predicted instances. A Panoptic Relational
Attention (PRA) module is then applied to the encodings and the global feature
map from the backbone. It produces a feature map that captures 1) the relations
across semantic classes and instances and 2) the relations between these
panoptic categories and spatial features. PISR also automatically learns to
focus on the more important instances, making it robust to the number of
instances used in the relational attention module. Moreover, PISR is a general
module that can be applied to any existing panoptic segmentation architecture.
Through extensive evaluations on panoptic segmentation benchmarks like
Cityscapes, COCO, and ADE20K, we show that PISR attains considerable
improvements over existing approaches.

本文提出了一种新的框架来整合语义和实例上下文以进行全景分割，该框架利用 Panoptic、Instance 和 Semantic Relations（PISR）模块来捕捉语义类别及预测实例的关键特征，在现有方法的基础上，通过引入关系关注机制以提高全景分割的性能。在 Cityscapes、COCO 和 ADE20K 等数据集上的实验表明，该方法相对于现有的方法具有显著的改进。

关系上下文编码器：增强全景分割的语义实例关系

Panoptic, Instance and Semantic Relations: A Relational Context Encoder  to Enhance Panoptic Segmentation

Modeling time-evolving knowledge graphs (KGs) has recently gained increasing
interest. Here, graph representation learning has become the dominant paradigm
for link prediction on temporal KGs. However, the embedding-based approaches
largely operate in a black-box fashion, lacking the ability to interpret their
predictions. This paper provides a link forecasting framework that reasons over
query-relevant subgraphs of temporal KGs and jointly models the structural
dependencies and the temporal dynamics. Especially, we propose a temporal
relational attention mechanism and a novel reverse representation update scheme
to guide the extraction of an enclosing subgraph around the query. The subgraph
is expanded by an iterative sampling of temporal neighbors and by attention
propagation. Our approach provides human-understandable evidence explaining the
forecast. We evaluate our model on four benchmark temporal knowledge graphs for
the link forecasting task. While being more explainable, our model obtains a
relative improvement of up to 20% on Hits@1 compared to the previous best KG
forecasting method. We also conduct a survey with 53 respondents, and the
results show that the evidence extracted by the model for link forecasting is
aligned with human understanding.

提出了一种基于关系注意力机制和反向表示更新策略的时间动态知识图谱链接预测框架，构建查询相关子图来预测图结构中未来的连接情况，具有更好的解释性和推荐准确率。

可解释的时间知识图谱推理用于未来链接预测

xERTE: Explainable Reasoning on Temporal Knowledge Graphs for  Forecasting Future Links

Describing images with text is a fundamental problem in vision-language
research. Current studies in this domain mostly focus on single image
captioning. However, in various real applications (e.g., image editing,
difference interpretation, and retrieval), generating relational captions for
two images, can also be very useful. This important problem has not been
explored mostly due to lack of datasets and effective models. To push forward
the research in this direction, we first introduce a new language-guided image
editing dataset that contains a large number of real image pairs with
corresponding editing instructions. We then propose a new relational speaker
model based on an encoder-decoder architecture with static relational attention
and sequential multi-head attention. We also extend the model with dynamic
relational attention, which calculates visual alignment while decoding. Our
models are evaluated on our newly collected and two public datasets consisting
of image pairs annotated with relationship sentences. Experimental results,
based on both automatic and human evaluation, demonstrate that our model
outperforms all baselines and existing methods on all the datasets.

该研究利用编码器 - 解码器结构和关系注意力等特征，提出了一种新的在两个图像之间生成关系说明的模型，并透过对新收集及公开的数据集进行实验，证明其比现有的各种基准线和方法都要好。