We present an empirical study of adapting an existing pretrained text-to-text
model for long-sequence inputs. Through a comprehensive study along three axes
of the pretraining pipeline -- model architecture, optimization objective, and
pretraining corpus, we propose an effective recipe to build long-context models
from existing short-context models. Specifically, we replace the full attention
in transformers with pooling-augmented blockwise attention, and pretrain the
model with a masked-span prediction task with spans of varying length. In terms
of the pretraining corpus, we find that using randomly concatenated
short-documents from a large open-domain corpus results in better performance
than using existing long document corpora which are typically limited in their
domain coverage. With these findings, we build a long-context model that
achieves competitive performance on long-text QA tasks and establishes the new
state of the art on five long-text summarization datasets, often outperforming
previous methods with larger model sizes. Our code has been released at
this https URL.

本文介绍了一项关于预训练模型适应长序列输入的经验研究，并提出一种构建长上下文模型的有效方法，包括采用池化增强分块注意力替换 transformers 中的全局注意力机制、采用不同长度的遮盖跨度预测任务、使用随机串联的短文档等。最终，研究人员成功构建出具有竞争性的长文本问答模型，并在五个长文本摘要数据集上取得了新的性能最高记录。