Efficient attentions have greatly improved the computational efficiency of
Transformers. However, most existing linear attention mechanisms suffer from an
\emph{efficiency degradation} problem, leading to inefficiencies in causal
language modeling and hindering their application in long-range language
models. This problem is more pronounced under language modeling with unbounded
contexts. In this paper, we propose \textbf{L}inear \textbf{A}ttention
\textbf{V}ia \textbf{O}rthogonal memory~(\shortname) to address these
limitations, achieving strong performance while maintaining linear complexity.
\shortname employs orthogonal decomposition to compress a context into a
fixed-size orthogonal memory while effectively minimizing redundancy within the
context. Given that orthogonal memory compresses global information, we further
dissect the context to amplify fine-grained local information. Additionally, we
embed the relative position encoding into \shortname to improve the
extrapolation ability. Experimental results show that \shortname greatly
improves the efficiency of the causal language model with the best
extrapolation performance and outperforms other efficient baselines. Further,
we endeavor to employ \shortname for unbounded language modeling and
successfully scale the context length to 128K.

通过使用正交内存（LAVO），我们提出了线性注意力方法的一种改进，通过正交分解将上下文压缩为固定大小的正交内存，同时最小化上下文中的冗余，并通过嵌入相对位置编码来改善外推能力。实验证明，LAVO 极大地提高了因果语言模型的效率，并在最佳外推性能上优于其他高效方法。

通过正交内存实现线性关注

Linear Attention via Orthogonal Memory

Vision transformers (ViTs) have found only limited practical use in
processing images, in spite of their state-of-the-art accuracy on certain
benchmarks. The reason for their limited use include their need for larger
training datasets and more computational resources compared to convolutional
neural networks (CNNs), owing to the quadratic complexity of their
self-attention mechanism. We propose a linear attention-convolution hybrid
architecture -- Convolutional X-formers for Vision (CXV) -- to overcome these
limitations. We replace the quadratic attention with linear attention
mechanisms, such as Performer, Nyströmformer, and Linear Transformer, to
reduce its GPU usage. Inductive prior for image data is provided by
convolutional sub-layers, thereby eliminating the need for class token and
positional embeddings used by the ViTs. We also propose a new training method
where we use two different optimizers during different phases of training and
show that it improves the top-1 image classification accuracy across different
architectures. CXV outperforms other architectures, token mixers (e.g.
ConvMixer, FNet and MLP Mixer), transformer models (e.g. ViT, CCT, CvT and
hybrid Xformers), and ResNets for image classification in scenarios with
limited data and GPU resources (cores, RAM, power).

提出了一种基于线性注意力机制的混合体系结构 ——Convolutional X-formers for Vision（CXV）。通过将 Quintic Transformer，Nyströmformer 和 Linear Transformer 等线性注意力机制代替二次注意力机制，来减少 GPU 使用。CXV 在有限的数据和 GPU 资源（核心，内存，功率）场景下，比其他的架构如 Token mixers（例如 ConvMixer，Fnet 和 MLP Mixer），变换模型（如 ViT，CCT，CvT 和混合 Xformers）以及 ResNets 等，更适用于图像分类任务。