The Transformer architecture has shown to be a powerful tool for a wide range
of tasks. It is based on the self-attention mechanism, which is an inherently
computationally expensive operation with quadratic computational complexity:
memory usage and compute time increase quadratically with the length of the
input sequences, thus limiting the application of Transformers. In this work,
we propose a novel Clustering self-Attention mechanism using Surrogate Tokens
(CAST), to optimize the attention computation and achieve efficient
transformers. CAST utilizes learnable surrogate tokens to construct a cluster
affinity matrix, used to cluster the input sequence and generate novel cluster
summaries. The self-attention from within each cluster is then combined with
the cluster summaries of other clusters, enabling information flow across the
entire input sequence. CAST improves efficiency by reducing the complexity from
$O(N^2)$ to $O(\alpha N)$ where N is the sequence length, and {\alpha} is
constant according to the number of clusters and samples per cluster. We show
that CAST performs better than or comparable to the baseline Transformers on
long-range sequence modeling tasks, while also achieving higher results on time
and memory efficiency than other efficient transformers.

基于自注意力机制的转换器架构，提出了一种使用代理符号的聚类自注意力机制（CAST），用于优化注意力计算并实现高效的转换器。CAST 通过减少复杂度从 O（N^2）到 O（αN），N 为序列长度，α 是按照聚类的数量和每个聚类的样本数而定的常数，实现了更高的时间和内存效率。

CAST: 使用替代标记的聚类自注意力以提高 Transformer 效率

CAST: Clustering Self-Attention using Surrogate Tokens for Efficient  Transformers

Models using structured state space sequence (S4) layers have achieved
state-of-the-art performance on long-range sequence modeling tasks. An S4 layer
combines linear state space models (SSMs), the HiPPO framework, and deep
learning to achieve high performance. We build on the design of the S4 layer
and introduce a new state space layer, the S5 layer. Whereas an S4 layer uses
many independent single-input, single-output SSMs, the S5 layer uses one
multi-input, multi-output SSM. We establish a connection between S5 and S4, and
use this to develop the initialization and parameterization used by the S5
model. The result is a state space layer that can leverage efficient and widely
implemented parallel scans, allowing S5 to match the computational efficiency
of S4, while also achieving state-of-the-art performance on several long-range
sequence modeling tasks. S5 averages 87.4% on the long range arena benchmark,
and 98.5% on the most difficult Path-X task.

介绍了基于深度学习、HiPPO 形式和线性状态空间模型的结构化状态空间序列（S4）层，以及在其基础上引入了一种新的状态空间层 ——S5 层。与 S4 层使用多个独立的 SSM 相比，S5 层使用一个多输入、多输出的 SSM，从而实现了更高的计算效率和性能。S5 在长距离序列建模任务上达到了最新的最高性能，并使最难的 Path-X 任务的准确率达到了 98.5%。