Transformers have surpassed RNNs in popularity due to their superior abilities in parallel training and long-term dependency modeling. Recently, there has been a renewed interest in using linear RNNs for efficient sequence modeling. These linear RNNs often employ gating mechanisms in the output of the linear recurrence layer while ignoring the significance of using forget gates within the recurrence. In this paper, we propose a gated linear RNN model dubbed Hierarchically Gated Recurrent Neural Network (HGRN), which includes forget gates that are lower bounded by a learnable value. The lower bound increases monotonically when moving up layers. This allows the upper layers to model long-term dependencies and the lower layers to model more local, short-term dependencies. Experiments on language modeling, image classification, and long-range arena benchmarks showcase the efficiency and effectiveness of our proposed model. The source code is available at https://github.com/OpenNLPLab/HGRN.

提出了一种具有遗忘门的分层门控递归神经网络(HGRN)模型，其中遗忘门受可学习值下界限制，使得上层能够建模长期依赖，而下层能够建模更局部、短期的依赖关系。通过在语言建模、图像分类和长距离竞技场测试中进行实验，证明了该模型的高效性和有效性。

层次门控循环神经网络用于序列建模