Recent studies on pre-trained language models have demonstrated their ability
to capture factual knowledge and applications in knowledge-aware downstream
tasks. In this work, we present a language model pre-training framework guided
by factual knowledge completion and verification, and use the generative and
discriminative approaches cooperatively to learn the model. Particularly, we
investigate two learning schemes, named two-tower scheme and pipeline scheme,
in training the generator and discriminator with shared parameter. Experimental
results on LAMA, a set of zero-shot cloze-style question answering tasks, show
that our model contains richer factual knowledge than the conventional
pre-trained language models. Furthermore, when fine-tuned and evaluated on the
MRQA shared tasks which consists of several machine reading comprehension
datasets, our model achieves the state-of-the-art performance, and gains large
improvements on NewsQA (+1.26 F1) and TriviaQA (+1.56 F1) over RoBERTa.

本文提出了一个基于事实知识补全和验证的预训练语言模型框架，探讨了共享参数下的两个学习方案，并在 LAMA 和 MRQA 等任务中验证了其能够提供更丰富的事实知识和实现最新的表现。