Modeling long texts has been an essential technique in the field of natural
language processing (NLP). With the ever-growing number of long documents, it
is important to develop effective modeling methods that can process and analyze
such texts. However, long texts pose important research challenges for existing
text models, with more complex semantics and special characteristics. In this
paper, we provide an overview of the recent advances on long texts modeling
based on Transformer models. Firstly, we introduce the formal definition of
long text modeling. Then, as the core content, we discuss how to process long
input to satisfy the length limitation and design improved Transformer
architectures to effectively extend the maximum context length. Following this,
we discuss how to adapt Transformer models to capture the special
characteristics of long texts. Finally, we describe four typical applications
involving long text modeling and conclude this paper with a discussion of
future directions. Our survey intends to provide researchers with a synthesis
and pointer to related work on long text modeling.

本篇论文综述了最新的基于 Transformer 模型的长文本建模技术，其中介绍了长文本模型的定义、如何满足输入长度限制和改进 Transformer 的架构以有效扩展最大上下文长度，以及如何适应长文本的特殊特性。并且描述了长文本建模的四个典型应用，并探讨了未来的研究方向。