Recent breakthroughs of pretrained language models have shown the
effectiveness of self-supervised learning for a wide range of natural language
processing (NLP) tasks. In addition to standard syntactic and semantic NLP
tasks, pretrained models achieve strong improvements on tasks that involve
real-world knowledge, suggesting that large-scale language modeling could be an
implicit method to capture knowledge. In this work, we further investigate the
extent to which pretrained models such as BERT capture knowledge using a
zero-shot fact completion task. Moreover, we propose a simple yet effective
weakly supervised pretraining objective, which explicitly forces the model to
incorporate knowledge about real-world entities. Models trained with our new
objective yield significant improvements on the fact completion task. When
applied to downstream tasks, our model consistently outperforms BERT on four
entity-related question answering datasets (i.e., WebQuestions, TriviaQA,
SearchQA and Quasar-T) with an average 2.7 F1 improvements and a standard
fine-grained entity typing dataset (i.e., FIGER) with 5.7 accuracy gains.

本文研究预训练语言模型（如 BERT）在零样本的事实推理任务中的知识捕获，提出一种有效的弱监督预训练目标，通过实体知识的显式融入，实现了对真实世界知识的建模，实验结果表明，该模型在答题、实体类型标注等任务上均优于 BERT。