Transformers achieve remarkable performance in several tasks but due to their
quadratic complexity, with respect to the input's length, they are
prohibitively slow for very long sequences. To address this limitation, we
express the self-attention as a linear dot-product of kernel feature maps and
make use of the associativity property of matrix products to reduce the
complexity from $\mathcal{O}\left(N^2\right)$ to $\mathcal{O}\left(N\right)$,
where $N$ is the sequence length. We show that this formulation permits an
iterative implementation that dramatically accelerates autoregressive
transformers and reveals their relationship to recurrent neural networks. Our
linear transformers achieve similar performance to vanilla transformers and
they are up to 4000x faster on autoregressive prediction of very long
sequences.

通过使用线性核特征图将自注意力表示为线性点积，并利用矩阵积的结合律特性将复杂性从 O (N^2) 降至 O (N)，我们的线性变压器实现比循环神经网络快 4000 倍，但在自回归预测上达到了与基本变压器类似的性能。

Transformers 是 RNN：具有线性注意力的快速自回归 Transformers

Transformers are RNNs: Fast Autoregressive Transformers with Linear  Attention

We introduce a family of multilayer graph kernels and establish new links
between graph convolutional neural networks and kernel methods. Our approach
generalizes convolutional kernel networks to graph-structured data, by
representing graphs as a sequence of kernel feature maps, where each node
carries information about local graph substructures. On the one hand, the
kernel point of view offers an unsupervised, expressive, and easy-to-regularize
data representation, which is useful when limited samples are available. On the
other hand, our model can also be trained end-to-end on large-scale data,
leading to new types of graph convolutional neural networks. We show that our
method achieves competitive performance on several graph classification
benchmarks, while offering simple model interpretation. Our code is freely
available at this https URL

本文介绍了一种多层图形核的家族，并建立了图卷积神经网络与核方法之间的新联系。利用核特征图形的序列表示图形数据，从而将卷积核网络推广到了图形结构数据，实现了高效的数据表示和训练。在多个图形分类基准测试中，该方法实现了竞争性的性能，同时提供了简单的模型解释。