Efficient attentions have greatly improved the computational efficiency of
Transformers. However, most existing linear attention mechanisms suffer from an
\emph{efficiency degradation} problem, leading to inefficiencies in causal
language modeling and hindering their application in long-range language
models. This problem is more pronounced under language modeling with unbounded
contexts. In this paper, we propose \textbf{L}inear \textbf{A}ttention
\textbf{V}ia \textbf{O}rthogonal memory~(\shortname) to address these
limitations, achieving strong performance while maintaining linear complexity.
\shortname employs orthogonal decomposition to compress a context into a
fixed-size orthogonal memory while effectively minimizing redundancy within the
context. Given that orthogonal memory compresses global information, we further
dissect the context to amplify fine-grained local information. Additionally, we
embed the relative position encoding into \shortname to improve the
extrapolation ability. Experimental results show that \shortname greatly
improves the efficiency of the causal language model with the best
extrapolation performance and outperforms other efficient baselines. Further,
we endeavor to employ \shortname for unbounded language modeling and
successfully scale the context length to 128K.

通过使用正交内存（LAVO），我们提出了线性注意力方法的一种改进，通过正交分解将上下文压缩为固定大小的正交内存，同时最小化上下文中的冗余，并通过嵌入相对位置编码来改善外推能力。实验证明，LAVO 极大地提高了因果语言模型的效率，并在最佳外推性能上优于其他高效方法。