Despite the recent observation that large language models (LLMs) can store
substantial factual knowledge, there is a limited understanding of the
mechanisms of how they acquire factual knowledge through pretraining. This work
addresses this gap by studying how LLMs acquire factual knowledge during
pretraining. The findings reveal several important insights into the dynamics
of factual knowledge acquisition during pretraining. First, counterintuitively,
we observe that pretraining on more data shows no significant improvement in
the model's capability to acquire and maintain factual knowledge. Next, there
is a power-law relationship between training steps and forgetting of
memorization and generalization of factual knowledge, and LLMs trained with
duplicated training data exhibit faster forgetting. Third, training LLMs with
larger batch sizes can enhance the models' robustness to forgetting. Overall,
our observations suggest that factual knowledge acquisition in LLM pretraining
occurs by progressively increasing the probability of factual knowledge
presented in the pretraining data at each step. However, this increase is
diluted by subsequent forgetting. Based on this interpretation, we demonstrate
that we can provide plausible explanations for recently observed behaviors of
LLMs, such as the poor performance of LLMs on long-tail knowledge and the
benefits of deduplicating the pretraining corpus.

通过研究大型语言模型在预训练过程中获得事实知识的机制，发现预训练数据量的增加并不显著提高模型获得和保持事实知识的能力，训练步骤与记忆遗忘和事实知识的泛化之间存在幂律关系，重复训练数据会导致遗忘加速，而使用更大的批量大小有助于提高模型的遗忘鲁棒性。事实知识在预训练过程中的获取是通过逐步增加每一步预训练数据中出现的事实知识的概率，但此提升会受到后续遗忘的影响。根据这一解释，我们对大型语言模型的表现提供了合理的解释，如对尾部知识的糟糕表现以及去重预训练语料库的好处。