Recent research shows that pre-trained language models (PLMs) suffer from
"prompt bias" in factual knowledge extraction, i.e., prompts tend to introduce
biases toward specific labels. However, the extent and impact of prompt bias
within the model remain underexplored. In response, this paper quantifies the
bias with various types of prompts and assesses their impact on different
benchmarks. We show that: 1) all prompts in the experiments exhibit
non-negligible bias, with gradient-based prompts like AutoPrompt and OptiPrompt
displaying significantly higher levels of bias; 2) prompt bias can amplify
benchmark accuracy unreasonably by overfitting the test datasets, especially on
imbalanced datasets like LAMA. Based on these findings, we propose a
representation-based approach to mitigate the prompt bias during inference
time. Specifically, we first estimate the biased representation using
prompt-only querying, and then remove it from the model's internal
representations to generate the debiased representations, which are used to
produce the final debiased outputs. Experiments across various prompts, PLMs,
and benchmarks show that our approach can not only correct the overfitted
performance caused by prompt bias, but also significantly improve the prompt
retrieval capability (up to 10% absolute performance gain). Our findings shed
new light on the underlying predicting mechanisms of prompt-based queries in
PLMs. Hopefully, our plug-and-play approach can be a golden standard to
strengthen PLMs toward reliable knowledge bases. Code and data are released in
this https URL

该研究量化了各种类型提示的偏差，并评估了其对不同基准测试的影响。通过提出一种基于表示的方法来缓解提示偏差，研究发现该方法不仅可以纠正由提示偏差引起的过拟合性能，还可以显著提高提示检索能力。