Transformers have surpassed RNNs in popularity due to their superior
abilities in parallel training and long-term dependency modeling. Recently,
there has been a renewed interest in using linear RNNs for efficient sequence
modeling. These linear RNNs often employ gating mechanisms in the output of the
linear recurrence layer while ignoring the significance of using forget gates
within the recurrence. In this paper, we propose a gated linear RNN model
dubbed Hierarchically Gated Recurrent Neural Network (HGRN), which includes
forget gates that are lower bounded by a learnable value. The lower bound
increases monotonically when moving up layers. This allows the upper layers to
model long-term dependencies and the lower layers to model more local,
short-term dependencies. Experiments on language modeling, image
classification, and long-range arena benchmarks showcase the efficiency and
effectiveness of our proposed model. The source code is available at
this https URL

提出了一种具有遗忘门的分层门控递归神经网络 (HGRN) 模型，其中遗忘门受可学习值下界限制，使得上层能够建模长期依赖，而下层能够建模更局部、短期的依赖关系。通过在语言建模、图像分类和长距离竞技场测试中进行实验，证明了该模型的高效性和有效性。