Recent studies of the computational power of recurrent neural networks (RNNs)
reveal a hierarchy of RNN architectures, given real-time and finite-precision
assumptions. Here we study auto-regressive Transformers with linearised
attention, a.k.a. linear Transformers (LTs) or Fast Weight Programmers (FWPs).
LTs are special in the sense that they are equivalent to RNN-like sequence
processors with a fixed-size state, while they can also be expressed as the
now-popular self-attention networks. We show that many well-known results for
the standard Transformer directly transfer to LTs/FWPs. Our formal language
recognition experiments demonstrate how recently proposed FWP extensions such
as recurrent FWPs and self-referential weight matrices successfully overcome
certain limitations of the LT, e.g., allowing for generalisation on the parity
problem. Our code is public.

通过研究具有线性化注意力的自回归 Transformer，即线性 Transformer（LT）或快速权重编程器（FWP），我们发现许多应用于标准 Transformer 的知名结果可以直接转移到 LTs / FWPs，并证明了最近提出的 FWP 扩展成功地克服了 LT 的某些限制，如在奇偶性问题上的泛化。