How much knowledge do pretrained language models hold? Recent research
observed that pretrained transformers are adept at modeling semantics but it is
unclear to what degree they grasp human knowledge, or how to ensure they do so.
In this paper we incorporate knowledge-awareness in language model pretraining
without changing the transformer architecture, inserting explicit knowledge
layers, or adding external storage of semantic information. Rather, we simply
signal the existence of entities to the input of the transformer in
pretraining, with an entity-extended tokenizer; and at the output, with an
additional entity prediction task. Our experiments show that solely by adding
these entity signals in pretraining, significantly more knowledge is packed
into the transformer parameters: we observe improved language modeling
accuracy, factual correctness in LAMA knowledge probing tasks, and semantics in
the hidden representations through edge probing.We also show that our
knowledge-aware language model (KALM) can serve as a drop-in replacement for
GPT-2 models, significantly improving downstream tasks like zero-shot
question-answering with no task-related training.

本文通过在预训练中引入实体信号，将知识意识融入到语言模型的预训练中，无需改变 transformer 体系结构、插入显式知识层或添加语义信息外部存储。实验证明，仅通过添加这些实体信号进行预训练，可以在 transformer 参数中装载更多的知识，从而提高语言建模精度，并在 LAMA 知识探测任务中获得事实的正确性，利用边界分析显示出隐藏表示中的语义，同时表明我们的知识感知语言模型（KALM）可以作为一个可抛弃替换 GPT-2 模型，显著提高了零 - shot 问题回答等下游任务的表现。