Language models retain a significant amount of world knowledge from their
pre-training stage. This allows knowledgeable models to be applied to
knowledge-intensive tasks prevalent in information retrieval, such as ranking
or question answering. Understanding how and which factual information is
acquired by our models is necessary to build responsible models. However,
limited work has been done to understand the effect of pre-training tasks on
the amount of knowledge captured and forgotten by language models during
pre-training. Building a better understanding of knowledge acquisition is the
goal of this paper. Therefore, we utilize a selection of pre-training tasks to
infuse knowledge into our model. In the following steps, we test the model's
knowledge retention by measuring its ability to answer factual questions. Our
experiments show that masking entities and principled masking of correlated
spans based on pointwise mutual information lead to more factual knowledge
being retained than masking random tokens. Our findings demonstrate that, like
the ability to perform a task, the (factual) knowledge acquired from being
trained on that task is forgotten when a model is trained to perform another
task (catastrophic forgetting) and how to prevent this phenomenon. To foster
reproducibility, the code, as well as the data used in this paper, are openly
available.

本文的目标是建立对知识获取过程的更好理解。我们利用一系列预训练任务注入知识到模型中，并通过测量模型回答事实性问题的能力测试其知识保留能力。我们的实验表明，掩盖实体和基于逐点互信息的相关跨度的原则性掩盖方法可以更好地保留事实知识，而遮盖随机顺序的词标记会导致模型遗忘更多的事实知识。我们的发现表明，类似于执行任务的能力，通过训练执行其他任务时获取的（事实）知识也会被遗忘，该文提供了防止这种现象的方法。