Hierarchically gated linear RNN (HGRN,Qin et al. 2023) has demonstrated
competitive training speed and performance in language modeling, while offering
efficient inference. However, the recurrent state size of HGRN remains
relatively small, which limits its expressiveness.To address this issue,
inspired by linear attention, we introduce a simple outer-product-based state
expansion mechanism so that the recurrent state size can be significantly
enlarged without introducing any additional parameters. The linear attention
form also allows for hardware-efficient training.Our extensive experiments
verify the advantage of HGRN2 over HGRN1 in language modeling, image
classification, and Long Range Arena.Our largest 3B HGRN2 model slightly
outperforms Mamba and LLaMa Architecture Transformer for language modeling in a
controlled experiment setting; and performs competitively with many open-source
3B models in downstream evaluation while using much fewer total training
tokens.

HGRN2 通过引入基于外积的状态扩展机制，在不引入额外参数的情况下，显著增加了循环状态大小，提高了语言建模、图像分类和长距离竞技场方面的性能。

HGRN2：具有状态扩展的门控线性循环神经网络

HGRN2: Gated Linear RNNs with State Expansion

The trade-off between performance and computational efficiency in
long-sequence modeling becomes a bottleneck for existing models. Inspired by
the continuous state space models (SSMs) with multi-input and multi-output in
control theory, we propose a new neural network called Linear Dynamics-embedded
Neural Network (LDNN). SSMs' continuous, discrete, and convolutional properties
enable LDNN to have few parameters, flexible inference, and efficient training
in long-sequence tasks. Two efficient strategies, diagonalization and
$'\text{Disentanglement then Fast Fourier Transform (FFT)}'$, are developed to
reduce the time complexity of convolution from $O(LNH\max\{L, N\})$ to
$O(LN\max \{H, \log L\})$. We further improve LDNN through bidirectional
noncausal and multi-head settings to accommodate a broader range of
applications. Extensive experiments on the Long Range Arena (LRA) demonstrate
the effectiveness and state-of-the-art performance of LDNN.

长序列建模中性能和计算效率之间的权衡成为现有模型的瓶颈，本文基于控制理论中具有多输入和多输出的连续状态空间模型（SSMs），提出了一种新型神经网络，称为线性动力学嵌入神经网络（LDNN）。通过对角化和解耦然后快速傅里叶变换（FFT）的两种高效策略，将卷积的时间复杂度从 O (LNHmax {L, N}) 降低到 O (LNmax {H, log L})，实现了 LDNN 在长序列任务中的少参数、灵活推断和高效训练。我们通过双向非因果和多头设置进一步改进了 LDNN，以适应更广泛的应用领域。对长距离竞技场（LRA）进行了大量实验证明了 LDNN 的有效性和最先进的性能。