Prompt-based probing has been widely used in evaluating the abilities of
pretrained language models (PLMs). Unfortunately, recent studies have
discovered such an evaluation may be inaccurate, inconsistent and unreliable.
Furthermore, the lack of understanding its inner workings, combined with its
wide applicability, has the potential to lead to unforeseen risks for
evaluating and applying PLMs in real-world applications. To discover,
understand and quantify the risks, this paper investigates the prompt-based
probing from a causal view, highlights three critical biases which could induce
biased results and conclusions, and proposes to conduct debiasing via causal
intervention. This paper provides valuable insights for the design of unbiased
datasets, better probing frameworks and more reliable evaluations of pretrained
language models. Furthermore, our conclusions also echo that we need to rethink
the criteria for identifying better pretrained language models. We openly
released the source code and data at this https URL.

本文探讨了基于问题提示的探测方法可能存在的偏见、不一致性和不可靠性，强调了通过因果干预来消除偏差的必要性，并提出了更好的数据集设计、探测框架和更可靠的预训练语言模型评估标准。