We introduce a new pretraining approach geared for multi-document language
modeling, incorporating two key ideas into the masked language modeling
self-supervised objective. First, instead of considering documents in
isolation, we pretrain over sets of multiple related documents, encouraging the
model to learn cross-document relationships. Second, we improve over recent
long-range transformers by introducing dynamic global attention that has access
to the entire input to predict masked tokens. We release CDLM (Cross-Document
Language Model), a new general language model for multi-document setting that
can be easily applied to downstream tasks. Our extensive analysis shows that
both ideas are essential for the success of CDLM, and work in synergy to set
new state-of-the-art results for several multi-text tasks. Code and models are
available at this https URL

该研究提出了一种新的预训练方法，针对多文档语言建模，包括两个关键思想：一是通过预训练多个相关文档集合，鼓励模型学习跨文档关系；二是通过引入动态全局注意力，改善最近的长距离 Transformer，以访问整个输入来预测掩码标记。 CDLM 是一个新的通用的多文档语言模型，可以轻松应用于下游任务。