Prompt learning is susceptible to intrinsic bias present in pre-trained
language models (LMs), resulting in sub-optimal performance of prompt-based
zero/few-shot learning. In this work, we propose a null-input prompting method
to calibrate intrinsic bias encoded in pre-trained LMs. Different from prior
efforts that address intrinsic bias primarily for social fairness and often
involve excessive computational cost, our objective is to explore enhancing
LMs' performance in downstream zero/few-shot learning while emphasizing the
efficiency of intrinsic bias calibration. Specifically, we leverage a diverse
set of auto-selected null-meaning inputs generated from GPT-4 to prompt
pre-trained LMs for intrinsic bias probing. Utilizing the bias-reflected
probability distribution, we formulate a distribution disparity loss for bias
calibration, where we exclusively update bias parameters ($0.1\%$ of total
parameters) of LMs towards equal probability distribution. Experimental results
show that the calibration promotes an equitable starting point for LMs while
preserving language modeling abilities. Across a wide range of datasets,
including sentiment analysis and topic classification, our method significantly
improves zero/few-shot learning performance of LMs for both in-context learning
and prompt-based fine-tuning (on average $9\%$ and $2\%$, respectively).

我们提出了一种空输入提示方法，通过使用来自 GPT-4 的多样化的空意义输入来为预训练语言模型的内在偏差校准提供基准，从而提高了零 / 少样本学习的性能。