Memory is one of the most essential cognitive functions serving as a
repository of world knowledge and episodes of activities. In recent years,
large-scale pre-trained language models have shown remarkable memorizing
ability. On the contrary, vanilla neural networks without pre-training have
been long observed suffering from the catastrophic forgetting problem. To
investigate such a retentive-forgetful contradiction and understand the memory
mechanism of language models, we conduct thorough experiments by controlling
the target knowledge types, the learning strategies and the learning schedules.
We find that: 1) Vanilla language models are forgetful; 2) Pre-training leads
to retentive language models; 3) Knowledge relevance and diversification
significantly influence the memory formation. These conclusions are useful for
understanding the abilities of pre-trained language models and shed light on
designing and evaluating new learning and inference algorithms of language
models.

对预训练语言模型进行了多种实验，发现预训练模型有助于保持记忆，而非预训练的神经网络具有遗忘的问题，并且知识相关性和多样性会显着影响记忆形成。这些结论有助于了解预训练模型的能力，同时也为语言模型的新学习和推断算法的设计和评估提供了启示。