Transformer models are revolutionizing machine learning, but their inner
workings remain mysterious. In this work, we present a new visualization
technique designed to help researchers understand the self-attention mechanism
in transformers that allows these models to learn rich, contextual
relationships between elements of a sequence. The main idea behind our method
is to visualize a joint embedding of the query and key vectors used by
transformer models to compute attention. Unlike previous attention
visualization techniques, our approach enables the analysis of global patterns
across multiple input sequences. We create an interactive visualization tool,
AttentionViz, based on these joint query-key embeddings, and use it to study
attention mechanisms in both language and vision transformers. We demonstrate
the utility of our approach in improving model understanding and offering new
insights about query-key interactions through several application scenarios and
expert feedback.

本文介绍了一种新的可视化技术，旨在帮助研究人员理解转换器中的自我注意机制，并演示了基于 query-key 嵌入的交互式可视化工具 AttentionViz，以改进模型的理解和提供关于查询 - 键交互的新见解。

AttentionViz: Transformer 注意力的全局视图

AttentionViz: A Global View of Transformer Attention

We present Graformer, a novel Transformer-based encoder-decoder architecture
for graph-to-text generation. With our novel graph self-attention, the encoding
of a node relies on all nodes in the input graph - not only direct neighbors -
facilitating the detection of global patterns. We represent the relation
between two nodes as the length of the shortest path between them. Graformer
learns to weight these node-node relations differently for different attention
heads, thus virtually learning differently connected views of the input graph.
We evaluate Graformer on two popular graph-to-text generation benchmarks,
AGENDA and WebNLG, where it achieves strong performance while using many fewer
parameters than other approaches.

本文介绍了 Graformer，一种新颖的基于 Transformer 的编码器 - 解码器体系结构，用于图形到文本的生成。使用我们新颖的图形自我关注，节点的编码依赖于输入图中的所有节点，促进了全局模式的检测。通过学习不同关注头的节点之间的权重，Graformer 实现了不同的连接视图，从而在 AGENDA 和 WebNLG 这两个流行的图形到文本生成基准测试中实现强大的性能。

通过相对位置对知识图谱进行文本生成的图结构建模

Modeling Graph Structure via Relative Position for Text Generation from  Knowledge Graphs

Forecasting high-dimensional time series plays a crucial role in many
applications such as demand forecasting and financial predictions. Modern
datasets can have millions of correlated time-series that evolve together, i.e
they are extremely high dimensional (one dimension for each individual
time-series). There is a need for exploiting global patterns and coupling them
with local calibration for better prediction. However, most recent deep
learning approaches in the literature are one-dimensional, i.e, even though
they are trained on the whole dataset, during prediction, the future forecast
for a single dimension mainly depends on past values from the same dimension.
In this paper, we seek to correct this deficiency and propose DeepGLO, a deep
forecasting model which thinks globally and acts locally. In particular,
DeepGLO is a hybrid model that combines a global matrix factorization model
regularized by a temporal convolution network, along with another temporal
network that can capture local properties of each time-series and associated
covariates. Our model can be trained effectively on high-dimensional but
diverse time series, where different time series can have vastly different
scales, without a priori normalization or rescaling. Empirical results
demonstrate that DeepGLO can outperform state-of-the-art approaches; for
example, we see more than 25% improvement in WAPE over other methods on a
public dataset that contains more than 100K-dimensional time series.

本文提出了一种能够全局认知并在本地进行校准的深度预测模型 DeepGlo, 它是一个混合模型，将全局矩阵因式分解模型与另一个能够捕捉每个时间序列及相关协变量的局部属性的时间网络相结合，证明了 DeepGlo 对于高维时间序列的预测要比其他方法更加准确。