Most language model pre-training frameworks concatenate multiple documents
into fixed-length sequences and use causal masking to compute the likelihood of
each token given its context; this strategy is widely adopted due to its
simplicity and efficiency. However, to this day, the influence of the
pre-training sequence composition strategy on the generalisation properties of
the model remains under-explored. In this work, we find that applying causal
masking can lead to the inclusion of distracting information from previous
documents during pre-training, which negatively impacts the performance of the
models on language modelling and downstream tasks. In intra-document causal
masking, the likelihood of each token is only conditioned on the previous
tokens in the same document, eliminating potential distracting information from
previous documents and significantly improving performance. Furthermore, we
find that concatenating related documents can reduce some potential
distractions during pre-training, and our proposed efficient retrieval-based
sequence construction method, BM25Chunk, can improve in-context learning
(+11.6\%), knowledge memorisation (+9.8\%), and context utilisation (+7.2\%)
abilities of language models without sacrificing efficiency.

将相关文档连接起来并使用内部文档因果掩模，可以提高语言模型的性能，并通过 BM25Chunk 方法改善语境学习、知识记忆和上下文利用能力。