Standard multi-task benchmarks are essential for developing pretraining
models that can generalize to various downstream tasks. Existing benchmarks for
natural language processing (NLP) usually focus only on understanding or
generating short texts. However, long text modeling requires many distinct
abilities in contrast to short texts, such as the modeling of long-range
discourse and commonsense relations, and the coherence and controllability of
generation. The lack of standardized benchmarks makes it difficult to assess
these abilities of a model and fairly compare different models, especially
Chinese models. Therefore, we propose a story-centric benchmark named LOT for
evaluating Chinese long text modeling, which aggregates two understanding tasks
and two generation tasks. We construct new datasets for these tasks based on
human-written Chinese stories with hundreds of words. Furthermore, we release
an encoder-decoder-based Chinese long text pretraining model named LongLM with
up to 1 billion parameters. We pretrain LongLM on 120G Chinese novels with two
generative tasks including text infilling and conditional continuation.
Extensive experiments show that LongLM outperforms similar-sized pretraining
models substantially on both the understanding and generation tasks in LOT.

为了评估和公平比较不同模型的长文本处理能力，我们提出了一个名为 LOT 的以故事为中心的基准测试，并发布了一个名为 LongLM 的编码器 - 解码器型中文长文本预训练模型，训练了两个生成任务和 120G 中文小说，LongLM 在 LOT 中表现出在不同理解和生成任务方面优异的性能，胜过同等规模的预训练模型。