Transformers are not suited for processing long document input due to its quadratically increasing memory and time consumption. Simply truncating a long document or applying the sparse attention mechanism will incur the context fragmentation problem or inferior modeling capability with comparable model size. In this paper, we propose ERNIE-DOC, a document-level language pretraining model based on Recurrence Transformers. Two well-designed techniques, namely the retrospective feed mechanism and the enhanced recurrence mechanism enable ERNIE-DOC with much longer effective context length to capture the contextual information of a whole document. We pretrain ERNIE-DOC to explicitly learn the relationship among segments with an additional document-aware segment reordering objective. Various experiments on both English and Chinese document-level tasks are conducted. ERNIE-DOC achieves SOTA language modeling result of 16.8 ppl on WikiText-103 and outperforms competitive pretraining models on most language understanding tasks such as text classification, question answering by a large margin.

提出了基于具有回归的Transformers的文档级语言预训练模型ERNIE-Doc，借助回顾性馈送机制和增强的回归机制，提高了其处理长文档数据的能力。实验证明，在英文和中文文档级任务上，ERNIE-Doc在文本分类和问题回答等任务上表现出比其他模型更优秀的语言理解能力。

ERNIE-Doc: 一种用于回顾性长文档建模的Transformer