Previous works show that Pre-trained Language Models (PLMs) can capture
factual knowledge. However, some analyses reveal that PLMs fail to perform it
robustly, e.g., being sensitive to the changes of prompts when extracting
factual knowledge. To mitigate this issue, we propose to let PLMs learn the
deterministic relationship between the remaining context and the masked
content. The deterministic relationship ensures that the masked factual content
can be deterministically inferable based on the existing clues in the context.
That would provide more stable patterns for PLMs to capture factual knowledge
than randomly masking. Two pre-training tasks are further introduced to
motivate PLMs to rely on the deterministic relationship when filling masks.
Specifically, we use an external Knowledge Base (KB) to identify deterministic
relationships and continuously pre-train PLMs with the proposed methods. The
factual knowledge probing experiments indicate that the continuously
pre-trained PLMs achieve better robustness in factual knowledge capturing.
Further experiments on question-answering datasets show that trying to learn a
deterministic relationship with the proposed methods can also help other
knowledge-intensive tasks.

该论文提出了一种基于确定性关系的预训练方法，利用外部知识库识别确定性关系并连续预训练语言模型以提高其在提取事实知识方面的鲁棒性，试图学习与之相关的确定性关系有助于知识密集型任务。