Large language models (LMs) are currently trained to predict tokens given
document prefixes, enabling them to directly perform long-form generation and
prompting-style tasks which can be reduced to document completion. Existing
pretraining pipelines train LMs by concatenating random sets of short documents
to create input contexts but the prior documents provide no signal for
predicting the next document. We instead present In-Context Pretraining, a new
approach where language models are pretrained on a sequence of related
documents, thereby explicitly encouraging them to read and reason across
document boundaries. We can do In-Context Pretraining by simply changing the
document ordering so that each context contains related documents, and directly
applying existing pretraining pipelines. However, this document sorting problem
is challenging. There are billions of documents and we would like the sort to
maximize contextual similarity for every document without repeating any data.
To do this, we introduce approximate algorithms for finding related documents
with efficient nearest neighbor search and constructing coherent input contexts
with a graph traversal algorithm. Our experiments show In-Context Pretraining
offers a simple and scalable approach to significantly enhance LMs'performance:
we see notable improvements in tasks that require more complex contextual
reasoning, including in-context learning (+8%), reading comprehension (+15%),
faithfulness to previous contexts (+16%), long-context reasoning (+5%), and
retrieval augmentation (+9%).

大型语言模型通过 In-Context 预训练，在处理涉及相关文档的任务时，能够显著提高性能，包括复杂的语境推理、长文本推理、检索增强等。

上下文预训练：超越文档边界的语言建模

In-Context Pretraining: Language Modeling Beyond Document Boundaries

We describe a new method for summarizing similarities and differences in a
pair of related documents using a graph representation for text. Concepts
denoted by words, phrases, and proper names in the document are represented
positionally as nodes in the graph along with edges corresponding to semantic
relations between items. Given a perspective in terms of which the pair of
documents is to be summarized, the algorithm first uses a spreading activation
technique to discover, in each document, nodes semantically related to the
topic. The activated graphs of each document are then matched to yield a graph
corresponding to similarities and differences between the pair, which is
rendered in natural language. An evaluation of these techniques has been
carried out.

本论文提出了一种基于图表达的文本相似性和差异性总结方法，使用 “spreading activation” 技术在两个相关文档中发现与主题语义相关的节点，并将两个文档的激活图匹配以获得相似和不同之处的图。