Self-attention-based models have achieved remarkable progress in short-text
mining. However, the quadratic computational complexities restrict their
application in long text processing. Prior works have adopted the chunking
strategy to divide long documents into chunks and stack a self-attention
backbone with the recurrent structure to extract semantic representation. Such
an approach disables parallelization of the attention mechanism, significantly
increasing the training cost and raising hardware requirements. Revisiting the
self-attention mechanism and the recurrent structure, this paper proposes a
novel long-document encoding model, Recurrent Attention Network (RAN), to
enable the recurrent operation of self-attention. Combining the advantages from
both sides, the well-designed RAN is capable of extracting global semantics in
both token-level and document-level representations, making it inherently
compatible with both sequential and classification tasks, respectively.
Furthermore, RAN is computationally scalable as it supports parallelization on
long document processing. Extensive experiments demonstrate the long-text
encoding ability of the proposed RAN model on both classification and
sequential tasks, showing its potential for a wide range of applications.

本文提出了一种新型深度学习神经网络 RAN，采用自注意力机制和递归结构相结合的方法，实现了在长文本处理中的可重复性操作，对文本进行建模提取全局语义信息，并能兼容顺序和分类任务。实验结果表明，RAN 模型具有出色的编码能力，适用于长文本语义的自动提取和处理。