We introduce Attention Free Transformer (AFT), an efficient variant of
Transformers that eliminates the need for dot product self attention. In an AFT
layer, the key and value are first combined with a set of learned position
biases, the result of which is multiplied with the query in an element-wise
fashion. This new operation has a memory complexity linear w.r.t. both the
context size and the dimension of features, making it compatible to both large
input and model sizes. We also introduce AFT-local and AFT-conv, two model
variants that take advantage of the idea of locality and spatial weight sharing
while maintaining global connectivity. We conduct extensive experiments on two
autoregressive modeling tasks (CIFAR10 and Enwik8) as well as an image
recognition task (ImageNet-1K classification). We show that AFT demonstrates
competitive performance on all the benchmarks, while providing excellent
efficiency at the same time.

本研究提出了 Attention Free Transformer (AFT) 框架，它采用了一种高效的元素级别的矩阵运算，不需要使用点积自注意力机制，通过引入位置偏移可以降低存储复杂度，同时实现了全局连接与本地化结合的空间权重共享。在 CIFAR10、Enwik8 和 ImageNet-1K 等任务上，实验表明 AFT 框架显示出优秀的性能和高效性。