In recent studies, linear recurrent neural networks (LRNNs) have achieved Transformer-level performance in natural language modeling and long-range modeling while offering rapid parallel training and constant inference costs. With the resurged interest in LRNNs, we study whether they can learn the hidden rules in training sequences, such as the grammatical structures of regular language. We theoretically analyze some existing LRNNs and discover their limitations on regular language. Motivated by the analysis, we propose a new LRNN equipped with a block-diagonal and input-dependent transition matrix. Experiments suggest that the proposed model is the only LRNN that can perform length extrapolation on regular language tasks such as Sum, Even Pair, and Modular Arithmetic.

最近的研究表明，在自然语言建模和长期建模方面，线性递归神经网络（LRNN）取得了与Transformer相媲美的性能，同时提供了快速的并行训练和恒定的推断成本。通过研究LRNN的潜在规则学习能力，我们在理论上分析了一些现有的LRNN，并发现它们在正则语言上存在一些限制。在分析的基础上，我们提出了一种新的LRNN模型，它配备了一个块对角线和输入相关的转移矩阵。实验证明，所提出的模型是唯一能够在正则语言任务（如求和、偶数对和模运算）中进行长度外推的LRNN模型。

线性循环神经网络中推进常规语言推理