Several recent papers have investigated the potential of language models as
knowledge bases as well as the existence of severe biases when extracting
factual knowledge. In this work, we focus on the factual probing performance
over unseen prompts from tuning, and using a probabilistic view we show the
inherent misalignment between pre-training and downstream tuning objectives in
language models for probing knowledge. We hypothesize that simultaneously
debiasing these objectives can be the key to generalisation over unseen
prompts. We propose an adapter-based framework, UniArk, for generalised and
consistent factual knowledge extraction through simple methods without
introducing extra parameters. Extensive experiments show that UniArk can
significantly improve the model's out-of-domain generalisation as well as
consistency under various prompts. Additionally, we construct ParaTrex, a
large-scale and diverse dataset for measuring the inconsistency and
out-of-domain generation of models. Further, ParaTrex offers a reference method
for constructing paraphrased datasets using large language models.

使用适配器框架 UniArk，通过简单方法进行广义和一致的事实知识提取，大幅提高了模型在领域外的概括能力和多种提示下的一致性。此外，构建了 ParaTrex，一个大规模且多样化的数据集，用于测量模型的不一致性和领域外生成，并提供了使用大型语言模型构建改写数据集的参考方法。