Transformer models have achieved remarkable results in a wide range of
applications. However, their scalability is hampered by the quadratic time and
memory complexity of the self-attention mechanism concerning the sequence
length. This limitation poses a substantial obstacle when dealing with long
documents or high-resolution images. In this work, we study the self-attention
mechanism by analyzing the distribution of the attention matrix and its
concentration ability. Furthermore, we propose instruments to measure these
quantities and introduce a novel self-attention mechanism, Linear Log-Normal
Attention, designed to emulate the distribution and concentration behavior of
the original self-attention. Our experimental results on popular natural
language benchmarks reveal that our proposed Linear Log-Normal Attention
outperforms other linearized attention alternatives, offering a promising
avenue for enhancing the scalability of transformer models. Our code is
available in supplementary materials.

通过分析自注意力机制的注意力矩阵分布和其专注能力，并引入一种新的自注意力机制（线性对数正态注意力），我们试图研究其与原始自注意力的分布和专注行为的模拟度量，实验结果在流行的自然语言基准测试中表明我们提出的线性对数正态注意力优于其他线性注意力替代方案，为提高 Transformer 模型的可扩展性提供了有前景的途径。