Previous literatures show that pre-trained masked language models (MLMs) such
as BERT can achieve competitive factual knowledge extraction performance on
some datasets, indicating that MLMs can potentially be a reliable knowledge
source. In this paper, we conduct a rigorous study to explore the underlying
predicting mechanisms of MLMs over different extraction paradigms. By
investigating the behaviors of MLMs, we find that previous decent performance
mainly owes to the biased prompts which overfit dataset artifacts. Furthermore,
incorporating illustrative cases and external contexts improve knowledge
prediction mainly due to entity type guidance and golden answer leakage. Our
findings shed light on the underlying predicting mechanisms of MLMs, and
strongly question the previous conclusion that current MLMs can potentially
serve as reliable factual knowledge bases.

本文对预训练掩码语言模型（MLMs）在不同抽取范式下的预测机制进行了研究，发现之前的良好表现主要归因于偏见提示和外部环境的贡献。实体类型引导和正确答案泄露是知识预测的改进方向。这些发现揭示了 MLMs 的预测机制，同时质疑了现有 MLMs 可以作为可靠事实知识库的结论。