Previous literature has proved that Pretrained Language Models (PLMs) can
store factual knowledge. However, we find that facts stored in the PLMs are not
always correct. It motivates us to explore a fundamental question: How do we
calibrate factual knowledge in PLMs without re-training from scratch? In this
work, we propose a simple and lightweight method CaliNet to achieve this goal.
To be specific, we first detect whether PLMs can learn the right facts via a
contrastive score between right and fake facts. If not, we then use a
lightweight method to add and adapt new parameters to specific factual texts.
Experiments on the knowledge probing task show the calibration effectiveness
and efficiency. In addition, through closed-book question answering, we find
that the calibrated PLM possesses knowledge generalization ability after
fine-tuning. Beyond the calibration performance, we further investigate and
visualize the knowledge calibration mechanism.

本文提出了一种简单轻量级的方法 CaliNet，用于校准预训练语言模型（PLMs）中所保存的事实性知识，试图解决当前存储知识的准确性问题， 这一方法通过知识探测任务的实验表明其效率及有效性，并且经过微调后， 其校准后的 PLM 具有良好的知识泛化能力，并且我们还进一步研究了知识校准机制。