Recent progress in pretraining language models on large textual corpora led
to a surge of improvements for downstream NLP tasks. Whilst learning linguistic
knowledge, these models may also be storing relational knowledge present in the
training data, and may be able to answer queries structured as
"fill-in-the-blank" cloze statements. Language models have many advantages over
structured knowledge bases: they require no schema engineering, allow
practitioners to query about an open class of relations, are easy to extend to
more data, and require no human supervision to train. We present an in-depth
analysis of the relational knowledge already present (without fine-tuning) in a
wide range of state-of-the-art pretrained language models. We find that (i)
without fine-tuning, BERT contains relational knowledge competitive with
traditional NLP methods that have some access to oracle knowledge, (ii) BERT
also does remarkably well on open-domain question answering against a
supervised baseline, and (iii) certain types of factual knowledge are learned
much more readily than others by standard language model pretraining
approaches. The surprisingly strong ability of these models to recall factual
knowledge without any fine-tuning demonstrates their potential as unsupervised
open-domain QA systems. The code to reproduce our analysis is available at
this https URL

通过对预训练语言模型的深入分析，我们发现未经微调的 BERT 模型竞争传统 NLP 方法的关系知识，可以根据开放式关系进行查询，某些类型的事实知识比标准语言模型预训练方法更容易学习，并可以作为无监督的开放式 QA 系统的潜力展现。