The design choices in the Transformer attention mechanism, including weak inductive bias and quadratic computational complexity, have limited its application for modeling long sequences. In this paper, we introduce Mega, a simple, theoretically grounded, single-head gated attention mechanism equipped with (exponential) moving average to incorporate inductive bias of position-aware local dependencies into the position-agnostic attention mechanism. We further propose a variant of Mega that offers linear time and space complexity yet yields only minimal quality loss, by efficiently splitting the whole sequence into multiple chunks with fixed length. Extensive experiments on a wide range of sequence modeling benchmarks, including the Long Range Arena, neural machine translation, auto-regressive language modeling, and image and speech classification, show that Mega achieves significant improvements over other sequence models, including variants of Transformers and recent state space models.

本文介绍了Mega，这是一种简单，从理论上得到支持的单头门控注意力机制，具有指数移动平均数以将位置感知的局部依赖性的归纳偏差纳入位置不可知的注意力机制中。通过将整个序列有效地分成多个具有固定长度的块以实现线性时间和空间复杂度，该文进一步提出了Mega的变体。在长序列建模、神经机器翻译、自回归语言建模以及图像和语音分类等广泛测试中，证明了Mega优于其他序列模型，包括Transformer的变体和最近的状态空间模型。

Mega: 移动平均装备的门控注意力